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sponsoring the publication of these Proceedings. Papers published in this volume were presented at the 
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sold in both hardcopy and electronic versions. Volumes | and 2 are available through the Educational 
Resources Clearinghouse (ERIC) System. Proceedings volumes are available to members at AECT.ORG. 
Proceedings copies are also available at: 
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The Proceedings of AECT’s Convention are published in two volumes. Volume #1 contains papers dealing 
primarily with research and development topics. Papers dealing with the practice of instructional 
technology including instruction and training issues are contained in Volume #2. This year, both volumes 
are included in one document. 


REFEREEING PROCESS: Papers selected for presentation at the AECT Convention and included in these 
Proceedings were subjected to a reviewing process. All references to authorship were removed from 
proposals before they were submitted to referees for review. Approximately sixty percent of the 
manuscripts submitted for consideration were selected for presentation at the convention and for 
publication in these Proceedings. The papers contained in this document represent some of the most current 
thinking in educational communications and technology. 
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Introduction 


Instructional design and technology (IDT) as field of study and practice does not formally exist in the 
Tunisian higher educational system. This is apparent due to the absence of such degree in the Tunisian academia as 
well as the absence of specialized units that are dedicated to support teachers/instructors/faculty (whether at schools 
or universities) with their IDT needs. 


Motivated to positively contribute to the educational reform in Tunisia and based on an expressed need— 
from a large regional university in Tunisia—for acquiring IDT skills, we (the authors) proposed to this university the 
idea of designing and delivering an IDT training to their faculty members. Funded by a local grant obtained from 
Indiana University, and by relying on the authors combined 20 years of experience in the IDT field, we designed, 
developed, and conducted a three-half day face-to-face training that targeted university faculty and instructors. After 
conducting the training, we conducted an evaluation study to investigative the effectiveness of the training and to 
determine its overall worthiness. In this paper we share the results of this evaluation study with the goal of 
informing the practice of international IDT communities. 


Review of Literature 


In this section we share highlights from the IDT literature that focus on IDT trainings in higher education 
contexts, and background information about the Tunisian higher education system. Key pieces of literature were 
identified through searching three online databases (Google Scholar, Academic Search Premier and Eric) and by 
applying the following selection criteria: peer-reviewed scholarly articles, books, books chapters, and reports. 


An IDT training is crucial for faculty to know how to apply IDTs in their content areas. Several studies in 
IDT literature centering on the topic of faculty training in the area of IDT indicate the importance of training and 
provide useful suggestions on how to ensure the success of such trainings. For instance, to answer many questions 
related to teaching and learning, Leh (2005) states that training is a necessity for higher education faculty. As far as 
access and use of technologies are concerned, IDT training is argued to be a prerequisite for providing technology 
support (Ali, 2003). In investigating what makes IDT training successful (i.e., how to make it work and/or achieve it 
desired goals?), several scholars pose this question to find effective training strategies (Georgina & Hosford, 2009). 
Personalized, sustained, and contextualized trainings are among the strategies that afford a training to be successful as 
‘one size does not fit all’ (de Vry 2003; Karlin, Ottenbreit-Leftwich, Ozogul, & Liao, In Press). 


There are around 266 higher education institutions in Tunisia, including private and public. (EACEA, 2017). 
Several higher education reforms in Tunisia have been introduced over the last three decades. These reforms were 
primarily aimed at implementing the “new maitrise” (Bachelor of Arts program) (Daoud, 1996) and the 
introduction—in 2006—of a new degree system called L.M.D—License (Bachelor) Mastere (Master’s), Doctorate 
(Ph.D.). In the L.M.D system, academic progress is measured in credit hours (similar to the U.S. system), and 4 years 
worth of coursework is condensed into a 3-year span of time (EACEA, 2017). 


Such reforms resulted in challenges for faculty to address their curriculum goals and implement their 
teaching methods due to the absence of IDT training opportunities and support structures (Lachheb, 2013). 
Considerable efforts are being made by Tunisian faculty and different higher education stakeholders to either change 
or abolish this system. 


Background of this Specific IDT Training 


Based on an expressed need for an IDT training from a regional large university in Tunisia, we gleaned 
information about the context of the training, potential attendees, and its overall logistics. Once we gathered all the 
information needed from the host university, we started drafting an outline of three workshops that formed the core of 
the training. Each workshop addressed the needs stated by the potential trainees, which were: (1) Basic introduction to 
instructional design: needs assessment and writing effective learning objectives, (2) Instructional methods, (3) Using 
free learning management systems. 


While we did not follow a specific ID model, we kept our ID process close to the Kemp Instructional Design 
Model (Morrison, Ross, & Kemp, 2012). This model is characterized by non-linear structure with the nine 
independent core elements, such as designing the message, instructional strategies, content sequencing, instructional 
objectives, task analysis, learner characteristics, instructional problems, evaluation instruments, and instructional 
delivery (figure 1). This model allows the design process to be flexible and responsive to ongoing adjustments and 
revisions. In this context, the aforesaid instructional design model enabled us to design a training that thoroughly 
addressed the expressed learning needs of potential trainees. In addition to the Kemp ID model, we relied on 
evidence-based principles of media design and instruction such as: Contrast, Alignment, Repetition and Proximity 
(CARP, Reynolds, 2011), and Gagne’s (1992) Nine Events of Instruction in structing the different sessions of the 
training. We developed PowerPoint presentations (figure 2), handouts (figure 3) as well as a Canvas online course to 
host the training Resources (figure 4). 
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Figure 1. Kemp ID Model (Morrison, Ross, Kalman & Kemp, 2012) 


Learning Objectives-The 
How? 


: <5. eapanitaamimnnal  saececmnanen aoe 
se elalalial fe] Creating coder poteet dent anantnunene 


Can the student justify a stand "Y appraise argue, defend, judge, select 
or decision? support, valve, evaluate 


Bloom’s Taxonomy (Revised) 


Evaluating 


appraise, compare, contrat, oibze, 


1. What do | want my target audience to be able Canthe student aitingush Y Shortie dacianae drys 


Analyzing, between different parts? “examine, experiment, question, test 
to do? \ 


2. What are the important conditions or Applying crt een 
constraints under which | want them to perform? ) 

3. How well must my target audience perform 
for me to be satisfied? 


last, describe, discuss, explain, 
entity locate, recogni, report 


Y) concepts? select, tranlate, paraphrase 
‘ Can the student recall or define, upkcate st, memoree, recall 
remember the information? repeat state 


irameraaee ‘Can the student explain ideas or 


Figure 2. Examples of slides from a PowerPoint presentation developed for this training (Day 1, session 2: writing 
effective learning objectives) 


First Time Teaching! 


Oumaima just graduated from a prestigious university with a PhD in English linguistics. 
She got hired by the University of Sousse to teach at their English department. Two weeks 
before the semester started her department chair, Dr. Chokou, requested a meeting with 
her to discuss her fall semester course load. During the meeting, Dr. Chokou told 
Oumaima that one of the faculty members relocated to work in Tunis, thus, disrupted his 
plans. She is now asked to teach English pronunciation lab for 1° year and 2" second 
year students, majoring in English Arts and Business English. 


This is the first time Oumaima will be teaching this class. She never received any training 
‘on teaching such course. Teaching business English is also new to her. The only thing 
she knows about this course was her experience as a student, 10 years ago. Yet, she 
was both nervous and excited to meet the new challenges. As Oumaima drove back to 
home after her meeting with Dr. Chokou, she remembered the conversation she had with 
him. Dr. Chokou mentioned how his faculty needed to use technology in teaching their 
courses, and it’s now a requirement, not an option. Failing to do so will reflect badly on 
her chances for long-term employment. He also mentioned that he department enjoys a 
good access to newly acquired ICT tools. When asked for further guidance about 
technology use, Oumaima was asked to meet and talk to another professor who had 
taught this course before, Dr. Sofiene Bigbro. 


Figure 3. An example of a handout (a case study) that was distributed to the faculty attending the training for a 
discussion activity. 
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Technology Integration: Using Learning Management Systems (LMS) 


Figure 4. A screenshot of the Canvas course site that was used for this training (trainee view) 


The training was implemented by conducting three half-day workshops. Each workshop lasted 2 hours in a 
conference room, equipped with a computer and a projector, and participants could bring their own laptops and get 
connected to the university’s WIFI network. At the end of each workshop, participants were asked to complete an 
assignment and submit it through the training Canvas course site. The training was delivered by one of the authors in 
person due to his familiarity with the Tunisian higher education context and the culture. The training was delivered 
in English with frequent translation to French, since two trainees exhibited limited English proficiency and were 
more familiar with key concepts in the French language, e.g., conception pédagogique; instructional design in 
English. 


Method of the Evaluation Study 


This evaluation study followed a mixed-method approach (Creswell & Clark, 2011) and aimed to answer 
the following questions: EQ1: How satisfied were the faculty with their training experience? Why or why not? EQ2: 
What are the faculty’s learning gains obtained from this training? 


Data collection 


To answer each of the above questions, we fist obtained IRB approval from our institution to ensure the 
evaluation study was ethically acceptable prior to starting it. We collected both qualitative and quantitative data by 
designing and deploying (an anonymous) survey questionnaire using Qualtrics (appendix 1). The survey included 23 
items where training participants were asked to (1) evaluate the content of the training, (2) evaluate the format of the 
training, (3) evaluate the quality of the training materials (4) provide suggestions for improvements, (5) share ideas 
as to what trainings they would like to attend in the future, and (6) rate their overall training experience. At the end 
of the survey, training participants took a 9-item quiz to assess their learning gains from the training. The above 
described two sections of the survey allowed us to answer two of our evaluation questions—faculty satisfaction with 
their training experience and their learning gains. 


Participants 


The participants were the 13 university level instructors and faculty members who attended the three-half 
day training to gain IDT skills. Eleven faculty members were from various subject areas in the English department 
(literature, history and civilization, discourse analysis, pedagogy courses, etc.) and two instructors were from two 
different areas in the French department (writing, literature, and history). Nine participants agreed to complete the 
survey (69% completion rate) and were offered a small token of appreciation for their time. 


Findings 
Content and Format of the Training 


Survey respondents were asked to rate the content of the training through multiple-choice, multiple-answer 
answer, and open-ended questions. Most of the respondents found the content to be very/somewhat interesting and 
engaging (87.5% and 12.5% respectively). 87.5% of the survey respondents found topics of the training to be 
appropriately challenging, and useful to their professional and personal lives (figure 5). Qualitative statements from 
survey respondents expressed that the content of the training was helpful, innovative, and informative (see table 1). 
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Figure 5. I believe the content/skills I learned in this training will be useful (check all that apply) 


Table 1 


Additional comments related to the training 


Nothing (X2) 


The training was very helpful and innovative in the Tunisian context. 

I think the ideas shared in the training are out of the box 

I find the training to be quite helpful and useful for the audience in question. 
It was engaging, informative and fun to attend. 

Designing needs analysis plan is an important part of this training 


No Comments 
interesting and very useful to us. I got new knowledge of how to design objectives and choose the right verbs. 
Knowing about the stages of instructional design helps us to improve our course design. We often do not follow them. 


The survey respondents gave positive feedback regarding the format of the training. The majority of the 
respondents found the length of each session and the content of the training to be appropriate (88.99% and 62.50% 
respectively). Regarding the amount of assignments/homework that trainees were asked to complete, all respondents 
found the amount of assigned homework to be just right (nor too much, nor too little), and be very helpful/somewhat 
helpful in applying the newly acquired IDT skills (figure 6). Most of the respondents expressed appreciation for the 
discussion activities as they found them to be very helpful/somewhat helpful in learning the content (75% and 25% 
respectively). The demonstration of actual tools, e.g. Canvas LMS, was found to be effective/somewhat effective in 
helping the trainees discover new instructional design tools (50% and 50% respectively). In answering the question 
regarding activities (from least to most engaging), the survey respondents indicated that group activities and trainer 
demonstrations were the most engaging (see table 2). 


62.50% 


Very helpful 


Figure 6. Perceptions of effectiveness of assigned homework in helping participants learn IDT skills 


Table 2 


Somewhat helpful 


Ranking of the various training activities implemented in training 


Not helpful 


1: Least Engaging 2 3 4: Most Engaging 
Instructor-led demonstration of 
technology tools 0.00% 0.00% 25.00% 75.00% 
Group activities/tasks to practice 
concepts/skills during workshops 0.00% 12.50% 12.50% 75.00% 
Instructor's explanations of 
concepts about IDT 0.00% 0.00% 37.50% 62.50% 
Open lab time to work on 
assigned projects while instructor 
helped you individually 0.00% 25.00% 37.50% 37.50% 


Quality of the Training Materials 


The survey respondents emphasized the high quality of the training materials. The quality of the visuals 
used in the materials was found to be of excellent quality (87.50%). When asked to indicate “how helpful did you 
find the training materials in helping you learn instructional technology and design skills?” , the survey respondents 
reported finding the materials to be very helpful/somewhat helpful (6 % and 33.33% respectively, figure 7). 


Very helpful Somewhat helpful! Not helpfu 


Figure 7. Perceptions regarding training materials helping participants learn IDT skills 


Areas for Improvement 


Through open-ended questions, the survey respondents indicated areas for improvements. When asked to 
state the thing they like the least about the training, the survey respondents commented on the timing of the training, 
internet connection and other aspects related to the training (table 3). Also, respondents indicated that the open lab 
time to work on assigned projects was the least engaging activity (see table 2 above). 


Table 3 


What did you like least about the training? 


internet connection is not good 

The number of the participants is not enough 

the amount of details, it is too much. 

the theoretical part of it maybe because I already had some background knowledge 

The group activities 

Internet connection was not very helpful 

Timing. It was the end of the academic year. Many people were busy preparing for the holidays. 


Overall Training Experience 


The majority of respondents (87.5 %, figure 8) stressed the instructor’s excellent preparedness and 
presentation skills. The demonstration of technology tools was the most engaging part of the training. That is, 75% 
of the respondents found instructor-led demonstration of technology tools to be the most engaging (see table 2 
above). That confirms that the content covered in the IDT training is relevant to the given academic context. Last, 
the majority of the survey respondents indicated their willingness to take/maybe take future IDT training (71% and 
29% respectively) with the most of them showing preference toward an online format (figure 9). 
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Figure 8. The quality of the instructor’s training skills were 
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Figure 9. Taking the same training in online course format would be 
Faculty’s Learning Gains Obtained from this Training 


At the end of the aforesaid survey, respondents were asked to take a quiz, consisting of nine questions (2 
possible points for each question, 18 total possible points). This quiz was designed with the aim of assessing the 
respondents’ learning gains from the IDT training. The mean score of respondents is 13.63 out of 18 possible points. 
Based on this score, it is possible to conclude that upon completion of the training, the respondents were able to 
define instructional design, needs analysis, explain the purpose of writing learning objectives, as well as describe 
each school of learning, e.g., behaviorism, cognitivism, and constructivism. The respondents exhibited a clear 
understanding of the purpose of writing learning objectives, as well as instructional strategies drawn from different 
learning theories. 


Discussion & Conclusion 


The study findings highlighted the positive outcome of the IDT training that were designed, developed and 
delivered in a large regional university in Tunisia. Based on the findings of this study, we are encouraged to propose 
more trainings in the future for other universities in Tunisia. Our study also suggests the following future 
considerations: (1) Being aware of the cultural context is a critical factor in designing and in conducting a successful 
training. Specific organizational cultures should take precedent among other considerations when designing and 
delivering IDT trainings (Brown, 2003; Ertmer, 2005; Mayo et al., 2005). (2) An emphasis on innovative 
technological tools (such as LMS tools) will be more helpful for a similar target audience due to their basic 
familiarity with general learning theory and sound pedagogical practices. This is evident based on the data gathered 
from the survey respondents. (3) Trainees should be demonstrated examples of meaningful technology integration 
relevant to their context. As shown in the data, trainees found the activities relevant to their context to be the most 
engaging for them. (4) In addition to informational sessions, similar trainings should include practical or hands-on 
activities encouraging trainees to apply newly acquired knowledge and skills. (5) A rigorous needs assessment will 
be a key step to take before designing an IDT training. Without understanding the context of the training and being 
responsive to the need of our target audience, our IDT training could have been a failed training attempt. 
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Appendix 1: Survey Questionnaire 


1- How helpful did you find the training materials in helping you learn instructional technology and design skills? 
e =6Very helpful 
e Somewhat helpful 
e §=Not helpful 


2-The length for each of session was 2 hours: 
e Too long for the given content 
e Appropriate duration for the given content 
e Too short for the given content 


3-The training spanned over 3 days (2 sessions a day): 
e Too long for the given content 
e Appropriate duration for the given content 
e Too short for the given content 


4-The quality of the visuals used in the training materials was: 
e = Excellent quality 
e Medium quality 
e Poor quality 


5-The quality of the instructor’s training skills were: 
e = Excellent quality 
e Medium quality 
e Poor quality 


6-In general, I found the content of the training to be: 
e © Very engaging/interesting 
e Somewhat engaging/interesting 
e Not interesting 


7-In general, I found the topics covered in the training to be : 
e = Too difficult 
e Appropriately challenging 
e Too easy 


8-Please provide any additional comments related to the training: 


9-Typically, you were asked to complete homework by the end of each day and a major project: 
e The amount of homework was too much 
e The amount of homework was about right 
e The amount of homework was too little 


10- How effective did you find the assigned homework in helping you learn IDT skills? 
e =©Very helpful 
e Somewhat helpful 
e §=6Not helpful 
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11- How helpful did you find the discussion activities during the training? 
e = Very helpful in learning the content 
e Somewhat helpful in learning the content 
e Not helpful in learning the content 


12- The practice of demo-ing practical technology solutions like Canvas LMS was: 
e Very effective in helping me improve my teaching practice 
e Somewhat effective in helping me improve my teaching practice 


e Not effective in helping me improve my teaching practice 


13-Using a scale of 1 to 4, where | is least engaging and 4 is most engaging, rank the following training activities: 


Pe __ 
Instructor led demonstration of technology tools Rene 
Instructor explanations of concepts about IDT Tee 


Group activities/tasks to practice concepts/skills during workshops ane 
Open-Lab time to work on assigned projects while instructor helped you individually Rigs 


14- I believe the content/skills I learned in this training will be useful (check all that apply): 
Professionally (career related) and Personally (non-career related) 

Only professionally 

Only personally 

Neither professionally or personally 


15-What were your expectations from this training? Please list 2: 


16-What did you like best about the training and why? 
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17-What did you like least about the training? 


18-Would you like to take additional IDT training? 


e §6Yes 
e Maybe 
e No 


19-Taking the same training in online course format would be 
e Helpful for me to learn more about instructional design and its relevance to my practice 
e Somewhat helpful for me to learn about instructional design and its relevance to my practice 
e =Not helpful for me to learn about instructional design and its relevance to my practice 


21. Were the topics presented at the training sufficient to provide you with a good understanding of instructional 
design is. Why or why not? 


22. Were the topics presented at the training showed the relevance to your practice? 
23-Please provide any additional comments related to the training: 
The following quiz is meant to gauge your retention and understanding of the material presented during the IDT 
training, which took place July 10-12. The quiz consists of 10 questions. 

1. Which statement best describes instructional design (ID)? 

a. ID primarily refers to educational psychology (learning and cognition). 

The process of making learning efficient, effective, and less difficult. 

c. ID focuses on the analysis of best teaching practices 
2. The purpose of instructional design (ID) is to (select all that apply) 

a. Help professors create visually appealing instructional materials 

b. Save time, money, and improve human performance 


c. Ensure seamless experience of using educational technologies in the classroom 


3. Needs analysis (NA), first step in instructional design process, is conducted in order to... (select all that 
apply) 


a. Collect data on learner characteristics and learning context 

b. Identify needs correctly to know what exactly needs to be designed 

c. Analyze the context of the program where the course will be offered and to what extent the course is 
needed to cover a skill or a knowledge gap in the program 


4. The most common needs analysis (NA) techniques are...(select all that apply) 
a. Anonymous survey of students (paper-based or online) 


b. Face-to-face interviews (structured or semi-structured) 
c. Checking previous achievement levels (grades or pre-test) 
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5. Upon completion of needs analysis, it is possible to know... (select two correct answers) 


a. Whether instruction is the only or not solution to improve human performance 
b. How many students enjoyed a particular course 
c. How the instruction needs to be improved 


6. Instructors/professors need to write learning objectives in order to ... 
a. Make their instructional materials more engaging 
b. Define the expected goal of a curriculum, course, lesson, or activity in terms of demonstrable skills or 
knowledge that will be acquired by a student as a result of instruction 
c. Activate students' prior knowledge about the topic 


7. What kind of verbs are used when writing learning objectives (select two correct answers): 
a. Actions verbs that describe mental activity: read, write, recite, identify, analyze, etc. 
b. Stative verbs that express a state rather than an action: feel, think, know, appreciate, etc. 
c. Verbs than can be measured: define, repeat, record, discuss, apply, etc. 


8. Which learning objective is written in the correct format... 


Given a lecture, students will be able to learn what discourse analysis is 

For one hour, students will be able to know what the illocutionary act is 

At the end of the course, students will appreciate postcolonial literature 

Based on the lecture, students will be able to explain the Noam Chomsky's theory of transformational 
grammar 


aesp 


9. Please match each school of learning with its correct description 


Behaviorism Instructor-centered approach; only observable behaviors explain learning; drill and practice 
strategies and rote memorization 


Cognitivism Cognition is what makes learning happen; the brain works like a computer; we learn differently 
depending on the type of input (Schema ->schemata->learning) 


Constructivism Student-centered approach; learning happens when people interact with each other or work 
together; discussion and problem-based approach 
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Abstract 


This paper intended to reveal the results of a study that explored whether integrating gamification into online 
learning Web sites can attract students’ interest toward using and spending time in these sites. The sequential 
exploratory mixed method study has not only shown attractiveness but also effectiveness of integrating gamification 
into these online learning sites. 


Keywords: Gamification, Open and Distance Learning, Motivation, Engagement 


Introduction 


Literature suggests that online and face to face learning result in equivalent outcomes for student in most 
higher education settings. Reporting on 16 meta-analyses, Bernard Borokhovski, Schmid, Tamim, and Abrami 
(2014) concluded that thousands of studies indicate that online learners and classroom learners succeed at equivalent 
rates at a wide variety of outcome measures. For decades, researchers have been investigated “what makes the 
difference?” For example, Wladis, Conway, and Hachey (2016) found that while students enrolled in online courses 
were more likely to drop out of college, online course outcomes had no direct effect on college persistence. 


The figures show that there is a big demand for open and distance learning (ODL) in Turkey same as all 
over the world. In 2016-2017 academic year, the number of students registered to distance learning programs was 
3,152,398 (47.1 percent of all students) while total number of higher education students was 6,689,185 (HEC, 2018). 
There is an important point the readers should know about Turkey that ‘open education’ and ‘distance education’ are 
legally considered as two different forms of education: the term Open Education is used for those institutions 
employ traditional open university strategies while Distance Education is used for those synchronous interaction 
based online learning programs (HEC, 2014). The big majority of distance learners are actually in open education 
programs (in 2016-2017, total 3,077,779 out of 3,152,398 were open education students). On the other hand, these 
open education programs are also considered as traditional ODL, which refers to exam preparation model that 
requires students to study the specifically prepared print-based materials with their own pace and take proctored 
face-to-face exams organized in certain times in a semester (Peters, 2003). In many ODL providers including 
Anadolu University, this print-based self-study is supported with several online support services via mainly a 
learning management system (LMS). These services include synchronous and asynchronous interaction 
opportunities with course facilitators and other students, interactive learning materials (video, multimedia, ebooks, 
audio books, etc.), online and offline trial exams, opportunities to join social student clubs, and so forth. However, 
the analytics (Anadolu, 2017) as well as a previous study (Hakan et al., 2013) have shown that only a small percent 
of the students were using these online services (according to Hakan et al. only 16.4% and according to analytics 
only 24.7%). On the other hand, studies such as Mutlu, Erorta, Kara, & Aydin (2005), have uncovered that those 
students engaged in online services got better scores at the exams than those who did not use these services. May be, 
the low completion (graduation) rates in ODL can be explained with this engagement problem. According to the 
2016-2017 figures, total 257,068 (8.1 percent) of the large body of open and distance learners were able to graduate 
from their programs (HEC, 2018). 
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Gamification is one of the tools can be used to motivate learners to engage in online learning (Werbach, 
2013). Gamification in general refers to use of game components into non-game environments. Awards, leader 
board, challenges, and batches are among the most often used game components for gamification. Coccoli, Iacono, 
and Vercelli (2015) classified game elements, which Werbach and Hunter (2012) describe the subset of the elements 
that characterize game design, putting in evidence only those useful in the gamification processes. They are listed in 
the following, ordered on the basis of the abstraction level from the specific design element: 


e Dynamics: the higher abstraction level. They include constraints, emotions, narrative, progression, 
relationships. ‘stp! 


oer 


een 


e Components: the instantiations of mechanics and dynamics. They can appear in the form of achievements, 
avatars, badges, boss fights, collections (of objects, badges), combat, content unlocking, gifting, 
leaderboards, level, points, quest (predefined challenges with objectives and rewards), social graph, team, 
virtual goods (game assets with perceived or real-money value). 


In education (both online or face-to-face) gamification means use of these components into learning 
environments. Studies such as Barata, Gama, Jorge, and Goncalves (2013), Dominguez, et al. (2012), Herranz, 
Colomo-Palacios, and Seco (2015), Sheth, Bell, and Kaiser (2012), suggest that gamification motivates the learners 
toward online learning and boost in interest and participation. However, the majority of the studies in the literature 
covers integration of gamification into small scale single courses, and there is a scarcity of research on integrating 
gamification into online learning program sites. 


The study proposed to explore the attractiveness and effectiveness of gamification into online learning 
program sites. More specifically, the study intended to seek the answers of the following questions: (1) Was there an 
increase in the number of students engaging online learning services and average time spend in these environments 
after the integration of gamification? (2) how did the gamification influence the students’ performance in the online 
courses? and (3) what did the learner think about integrating gamification into web sites? 


Method 


A sequential exploratory mixed design was employed, and it was conducted in one of the largest 
foundation-based higher education institutions (semi-governmental, semi-private institutions) located in the West 
side of Turkey. It was originally planned to be conducted in Anadolu University (the open university of Turkey) but 
due to the limitations of the LMS and the security measures of Anadolu, it was quite difficult to do. So that an 
institution offering ODL programs and flexible to include gamification was chosen. It has around 12000 students 
and all are required to take core courses offered completely online. The instructional strategy is same as traditional 
ODL, exam preparation model in these core courses. For this study, several game components (a leader board, 
earning points, batches and levels) were integrated into the program web site and the courses. Data collected via 
computer logs, exam scores and semi-structured interviews. 


Game elements are the toolbox of gamification which includes all the different components like points, 
leaderboard, levels, badges, and challenges/achievements that can be put together in different combinations to make 
up different game systems. In this study, we integrated following game elements into the courses and the program 
site: 


e = Points - In this study, “Learning Point” was used as the first element. These quantify the player’s progress. 
In this research, players can earn points by downloading learning materials, watching videos, posting in 
discussion boards, and completing practice tests. 

e Leaderboard - This is a list showing the ranking of students according to their learning points they 
collected. The leaderboard will be updated weekly, enabling the players to monitor their rankings. The 
leaderboard is specific for the courses. For example, a learner might take a place in UFND course’s 
leaderboard with the points that he/she collects in this specific UFND course. 

e Badges - Badges define the individual’s performance by symbolizing desired outcomes in the game 
(Abramovich, Schunn & Higashi, 2013). In this design, badges serve the mechanics of competition of an 
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achievement. Badges are employed for different purposes. They can be used for setting goals, providing 
explanations about learning activities, identifying players who have shared experiences, providing them 
with status and giving them the right to brag (Antin & Churchill, 2011). This design uses them to set goals, 
impart status and give the right to brag. We think to give out badges during the semester and at its end. For 
example, the student who got the fullest points was given the badge of “Ready to Teach” at the end of the 
semester. 

e Levels - These are the features been used to categorize the students. There were 4 levels used in the design. 
They can level up based on the learning points they collected. 


Findings 
The reporting of the findings was organized into two sections. Quantitative findings about learners’ access 


to various learning activities in the course in pre- and post-gamification situations were summarized in Table 1. 


Table 1. Investigation of the learners’ access behaviors before and after gamification. 


Measurement Situation n X S sd t ; 

Access - Site Before 294 5,83 10,79 293 -12,75* 0,62 
After 294 17,99 20,07 

Access - Content Bere as 13,84 28,19 293 =9°35" 0,48 
After 294 47,43 64,62 
Before 294 3,74 9,62 293 -2,35** 0,14 

Access — e-Book TAer 1964 S16. aa 

Access - Test Before 294,89 412 293 -17,32* 0,71 
After 294 9,83 6,56 


The average of access to the e-learning sites before the gamification was X =5.83, while the average 
increased to X =17.99 after the gamification, and this difference found statistically significant. This finding suggests 
that integration of gamification into the e-learning sites has a significant impact on the students to access to the e- 
learning sites. 


Also, it was found that there was a statistically significant increase in pre-gamification and post- 
gamification situations in terms of students’ access to the content in e-learning sites. While the average of content 
access in a pre-gamification e-learning environment was X = 13.84, the average after gamification increased to 
X =47.43. This finding suggests that integration into the e-learning environment has a significant impact on 
students’ content access behaviors in the environment. 


Then, in terms of access of reading resources in e-learning environment, it was found that there was a 
statistically significant increase between pre-gamification and post-gamification situations. While the average of 
access to the reading resources in e-learning environment before the gamification was X = 3.74, while the average 
after the gamification has increased to X = 5.16. This finding shows that integration of gamification into the e- 
learning sites has a significant impact on students' access to the reading resources in the learning sites. 

Last but not least, it was found that there was a statistically significant increase between pre-gamification and post- 
gamification situations in terms of students’ access to the tests in e-learning sites. While the average of access to the 
tests in pre-gamification was X =1.89, while it increased to an average of X =9.83 after the gamification. This 
finding shows that integration of gamification into the e-learning sites has a significant impact on students’ access to 
the tests. 


On the other hand, the Pearson’s Correlation Analysis has shown that there is no meaningful relationship 
regarding to the learners’ academic performances in pre- and post-gamification situations. 
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Table 2. Pearson’s Correlation Coefficient related to academic success in pre- and post-gamification situations 
(n=69). 


n r p 
Learners’ Grades 69 0,178 0,14 


The semi-structured interviews with the participation of 16 students have shown that the gamification 
motivated them to enter the course sites more often than before. Almost all the students mentioned that they entered 
the system only once during the first half of the course (first 7 weeks before the gamification intervention started) 
but during the second half they visited the sites at least twice in every week. Gamification did not only increase the 
frequency of visits but also the learners’ satisfaction with the courses. Every student using the system and 
participating in the semi-structured interview stated that they were satisfied with the system. The following quotes 
represents the learners’ satisfaction and shared with different phrases by all the interviewees: 


“So, in fact, it's a nice and competitive system. It was different from normal UFND courses. That's why I like it. I 
enjoyed seeing my name on the leaderboard.” 18 years old Female, studies Civic Engineering. 

“At the beginning I did not give any attention to these game-like things; but later, after earning some points, started 
to talk with my classmates about these points and so forth, it, kind of, motivated me to do more in the courses and I 
did. I was fun and at the end I finished the courses successfully” 19 years old Male, studies Business Management. 


During the analysis of the interviews, it was noticed that some students had some problems about how 
game elements work, such as hot to collect points, and earn a badge. Here is a quota shared by a number of 
interviewees: 

“Obviously I didn't quite understand how to earn a badge. I won sometimes, didn't other times. I don't know exactly 
but I followed the instructions that should be done in general.” 18 years old Fine Arts student. 


It was also noticed that the learners valued/preferred the Leaderboard and Badges more than the points and 
levels. Even those students whose name had never appeared on the Leaderboard liked the idea of having such a 
board. They found it very motivating. Five interviewees mentioned that although their name was not on the board, 
they worked visited the site, downloaded the materials, watched the videos and completed the activities more than 
usually they do in their classes to be able to get more points to appear in the board. And those 3 students whose 
name were on the board mentioned that they did not need any other motivational elements, such as badges, etc., and 
seeing their names on the board were enough. The students also stated that earning awards was fine but did not 
really motivated them as much as badges and leaderboard. 


Conclusions 


The results suggest that there is a statically significant increase in distance learners’ actions before and after 
the integration of game elements into the ODL program sites, including visiting the course webpage, clicking the 
content, downloading, and completing course assignments. The results also show that there is not any secret recipe 
for using game elements in a big scale open and distance learning system. The key for success is the correct 
selection of game elements. Different researchers use different game element combinations and most of them found 
positive results. For this reason, researchers or practitioners should be careful about integrating game elements into 
their systems because they are not just tools. In our case, we found out that the learners preferred the leaderboard 
and badges. The results regarding leaderboard might be related to competitive nature of the educational system in 
Turkey. Learners especially in K12 find themselves in a culture and climate that values test-preparation and 
competition. So, learners who came from this kind of a learning culture may easily value similar activities such as 
leaderboard. Similarly, the badges can be related to a trend observed among higher education students not only in 
Turkey but also all around the world toward earning as much certificate as possible while they are pursuing their 
regular degrees. The participant students might feel a similarity between badges (as micro-credentials) and 
certificates, and so that, they preferred badges rather than other elements. 

As aresult, we believe that gameful approaches in education have broad potential to reframe formal 
education, encouraging student engagement, and ultimately leading to deeper and better learning. As the momentum 
for gameful learning environments increases, it is crucial to develop empirically informed design principles and 
guidance for implementation so that this approach can be applied across multiple learning environments. 
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Abstract 


As learners and educators embrace VR as an educational technology tool, and as this technology is 
anticipated to become ubiquitous in education and training, it is important to understand how culture affects 
learners’ interacting with it. Educational technology is generally not free of cultural values; intentional or not. These 
cultural values affect how learners, instructors and content creators interact with educational technology. This 
present research reviews studies on the impact of culture on the deployment of VR as an educational tool. Culture 
influences content design, acceptance and use by instructors and learners who interact with this technology. This 
review suggests that VR content creators be conscious of cultural biases when creating content. 


Keywords: Culture, diversity, Virtual Reality, educational technology, accessibility. 


Virtual Reality (VR) has been used in education and training because it stimulates and immerses learners in 
content so that they can grasp content faster and participate in their own learning. This immersion increases 
engagement with content, increasing opportunities for learning. As Freina and Ott (2015) have pointed out, VR 
hardware, including eyeglasses and other Head Mounted Displays (HMD), produce in the user a visceral feeling of 
being in the simulated world. This immersion appeals to the basic human senses, especially sight and sound, which 
may enhance learning. 

This immersion in content often leads to an understanding of and grappling with content. In preparing this 
content, the VR content creators may unwittingly transmit their cultural values and beliefs to the learners. This may 
be in the form of the learning objects and artifacts contained in the structure of the VR program. It could also be in 
the form of the references in the virtual world that the user interacts with, or the prerequisite cultural knowledge a 
user is expected to have so as to successfully interpret and respond to stimulus from the virtual world. 


Understanding Culture 


Baldwin, Faulkner and Hecht (2006) identified culture as consisting of a set of elements such as ideas, 
behavior and others that are shared by people in a social structure. For this review, Garcia and Dominguez’ (1997) 
definition of the characteristics of culture is used. They note that culture provides the lens through which we view 
the world. It is shared by members of a group, it considers cultural values that persist, even though people who 
adhere to them may not express them consistently. They note also that culture is a dynamic process, likely to change 
over time; and it provides the basis for childhood experiences through which children are socialized to the norms, 
values, and traditions of the cultural group. Culture, therefore affects how people think and do things; it can 
determine whether technologies are accepted or rejected by learners and instructors. Learners’ culture can therefore 
determine whether these learning technologies are successfully integrated in schools. Further, culture can also 
influence peoples’ persistence in interacting with technology. Often, too, content creators may, whether knowingly 
or unknowingly, transmit their preferred cultural values through educational technologies. Additionally, the learning 
medium (in this case, VR) also enables users to create their own culture within the medium. Learners become part of 
the virtual learning experience, and like a tribe, they adopt new cultural norms with which they interact. This paper 
reviews research that examines the impact of culture on VR as an educational technology. 
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Research question and methodology 


This research is guided by the research questions: To what extent does culture impact learners who use VR 
technology? To what extent do educational VR content creators consider the effects of culture when users interact 
with their content? 

This research used a review of literature to identify studies on culture and educational technology, 
especially VR educational technology. This research used a variety of articles sourced from the major educational 
databases including ERIC, EBSCO, ProQuest, PsycINFO, among others. Research was also sourced from AECT’s 
TechTrends and Google Scholar. 


Culture in VR content 


Virtual Reality, as a technological tool, by itself does not possess any cultural or social attributes. It is a 
tool employed by users and its creators to achieve certain objectives — including entertainment, education, training, 
creating a virtual experience for safety, mining, tourism, among others. Instead, as Lanier and Biocca (1992) have 
shown, VR’s cultural potential can be realized by cultural developments that spring up alongside it. These include 
cultural symbols, meanings, and objects with which content creators imbue VR technology. Additionally, users of 
VR technology can also alter and re-purpose VR content so that it aligns with their cultural preferences. Lanier and 
Biocca point out that, like all technology, VR can be used as a route to access a community, to influence their 
experiences and it can also be a means of social communion, just like the TV and radio before it. 

Often the tools with which we interact, can alter our behavior. VR educational tools, like other tools, have 
influenced the way we interact with learning content. Rosenfeld (2015) notes that the affordances of the 
technological tools enables and inhibits individual agency which often push or pull us into uncharted ways of 
behaving. She finds that computers contain identity embedded within their database design, their user interface 
design and their functionality. This is also true for VR educational technology where artefacts, identities and realia 
are constructed to represent and mimic real-life. Learners’ interactions with these technologies are linked to cultural 
experiences that hold a powerful appeal to their psyche (Rosenfeld, 2015). 

McLoughlin and Oliver (2000) note that culture pervades learning, and in designing instructional 
environments, that there needs to be serious debate about issues concerning the social and cultural dimension of task 
design. These would be important in structuring learning goals that meet the needs of culturally diverse learners. 
McLoughlin and Oliver note that educational technology is imbued with cultural values and assumptions and that as 
a ‘cultural amplifier’ it can quantitatively change the processes of cognition of the users. 

Additionally, the users’ cultural background may affect how they interact with VR. In an experiment to 
investigate the role played by culture in the emotional responses to VR, Gorini et al (2009) selected two samples of 
Mexican participants undergoing ambulatory surgery to interact with VR. One group consisted of inhabitants of a 
rural isolated Mexican village, with elemental cultures, while the other second group were inhabitants of civilized 
culture of Mexico City. The research found that, depending on the users’ cultural and technological backgrounds, 
users have different emotional responses to interaction with VR. This research suggested that users’ cultures can 
fundamentally influence how they interact with VR. Morgan (2002) points out the importance of VR content 
designers in matching the content to the culture of the intended target audience. He underlines previous research that 
has suggested that previous exposure to technology resulted in learners being able to quickly adopt learning via VR 
technologies. To lessen the impact of cross-cultural differences in interacting with digital content, Morgan suggests 
the introduction of cross-cultural design teams and debriefing sessions to determine some of the cross-cultural 
learning needs of the target population. 


Positive effects of culture in VR technology 


Other researchers have shown how the culturally-infused aspects of VR have led to accelerated gains by 
learners. O’Brien and Levy (2008) note that in a language learning class, the affordances of VR to include cultural 
paraphernalia of the language that learners intend to learn provides a conducive setting for foreign language learners. 
Therefore, by learning a language in a virtual reality context that is infused with the culture of the language that they 
want to learn, greatly assists in faster language acquisition. This cultural experience through VR affords language 
learners contextualized practice with everyday language as experienced by the native speakers. O’Brien and Levy 
note that VR, unlike other multimedia, affords language learners the advantage to physically experience culture and 
interact with language in a virtually authentic environment that enhances cognition and retention of language. They 
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report that by using VR in learning language, students were successful in learning because VR enabled them to 
experience the target culture in a new, involving and meaningful way. 

Additionally, on the affordances of VR to boost language acquisition, Schwienhorst (2002) notes that VR 
can contribute toward language and linguistic awareness, while providing a more stress-reduced and egalitarian 
learning environment or collaboration and interaction between peers. In this way, learners who are still developing 
some degree of learner autonomy can benefit more from a VR environment where they are encouraged to 
communicate, collaborate and participate in the learning process. This process encourages them to communicate, 
collaborate, and participate in the learning process which encourages and sometimes forces them to take control of 
their own learning (Schwienhorst). 

Other researchers (for example; Noh, Sunar and Pan, 2009) have suggested that VR learning systems that 
can transmit cultural artefacts and symbols have been instrumental in reconstructing cultural architectural structures. 
The ability of VR to transmit cultural values has also been invaluable in spreading religious practices (Wagner, 
2012). Due to the ability of VR to recreate realistic circumstances, religious scholars can reconstruct scenes from the 
holy books that represent key religious tenets. VR is thus able to recreate religious rituals, stories and rich 
interactivity that invite immersion in experiences that enrichen religious cultural practices. 

Since most educators, learners, and users of VR have neither the technical skill nor the time to create their 
own VR content, they rely on content created by third parties. It is important therefore that these third-party content 
creators ensure that content is flexible so that the final users can customize it to suit their local conditions and 
cultures. VR content designers should also be aware of their cultural biases as they design content. 


VR technologies altering user perspectives 


Other researchers have noted that the recreated worlds afforded by VR alter the way a user perceives 
herself as she interacts with these virtual environments. Hillis (1999) suggested that virtual environments afforded 
by VR technology, fragment peoples’ identities and cultures, even as they promote a new form of determinism and 
mystical thinking. This researcher suggested that the creators of VR technology often promote it as superior to the 
cultural restrictions that constrain users in their regular lives and that VR environments free them from these 
constraints. VR environments change the way users perceive their world; the users can regulate this world to their 
liking, they alter their perception of relationships with objects and how to interact with them. 

Additionally, VR learning environments attempt to create a co-presence with users to achieve believability. 
Although, as for now, VR learning environments cannot replace the co-presence of a face-to-face instructor, VR 
creators try to create an ambiance that mimics real-life interaction. This attempt to create a co-presence requires that 
the user and the digital environment share a common culture. VR may not offer physical co-presence, but it can 
offer linguistic co-presence that requires the user to be attuned to the culture of the instructor’s co-presence. In this 
way, VR virtual environments fundamentally contribute to an alteration of the user’s culture to align it with the 
content creator’s culture, even if temporarily. Schwienhorst (2002) notes that this need for co-presence often results 
in language and linguistic awareness that provides a more egalitarian learning environment which is conducive for 
collaboration. VR learning environments seek to homogenize the learning environment so that as many users as 
possible may interact with it. Often, this means that the specific cultures of individual learners are thus subsumed to 
reflect the general cultural environment of the co-learners and to match the co-presence of the learning environment. 
This learning environment promotes common linguistic reference points (Schwienhorst) that may adhere to the 
culture of the content creator. Yee, Bailenson, Urbanek, Chang, & Merget (2007) reiterate this point when they note 
that social interactions in online virtual environments, including VR environments, are governed by the same social 
norms as social interactions in the physical world. The way users interact with and manipulate their VR avatars 
could also point to how they physically interact with similar objects in real life. 


Discussion: Why is this important? 


As this review suggests, several researchers have shown that VR is impacted by culture in several ways. 
The content of the VR system, its learning environments, and learners who interact with it are influenced by culture 
and they in turn also assert some cultural effect on these systems and environments. However, technologies and 
learning advantages afforded by VR systems remain powerful and provide an alternative means to traditional forms 
of learning. Some researchers even caution administrators and others not to be overly concerned by the cultural 
impact of VR on learners. Gayol and Schied (1997), for example, note that often cultures look for an abstract 
‘enemy’ outside the boundaries of cultural values and those who embrace such ideologies fear that their cultures will 
be polluted, and that such ‘cultural contamination’ should be avoided. However, they note, the introduction of VR in 
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society and other modern digital media, serves to break down the attempt to enclose people and societies in fixed 
clusters. 

Ultimately, cultural values seep into VR technology and affect users’ interaction with it. One of the main 
effects of culture on VR include the use of audio which contains language. Language is a definitive expression of 
culture; it governs how groups interact and communicate with each other. The language in VR technologies is 
determined by the content creators and it is an expression of culture. VR technologies are also affected by symbols 
which can be interpreted differently by users depending on their cultures. Some words, phrases, idioms and other 
language expressions, are also subject to misinterpretation across VR learning cultures. Additionally, the images and 
symbols employed in VR content could also impact learners depending on their cultures. Religious symbols, 
symbols of warfare and violence, politics, clothing, gestures, handedness, and colors (Morgan, 2002) largely affect 
users and they way they interact with VR technologies. 


Future projections 


There is no denying that technology, including VR technology, will continue to be integrated in education 
and training. More and more learners are continuously being exposed to digital media and are developing a greater 
preference for digital technologies and social media than previous generations (Oh and Reeves, 2014). Technology, 
including VR, will in the future continue to play a central role in the lives of learners, transforming the way they 
learn. Apart from being used for entertainment and amusement, VR will be used by learners of all ages and in 
varying fields. 


Conclusion 


VR, as a learning technology, is slowly achieving acceptability and an increase in usage. Consumer 
demand for VR headsets are expected to reach 37 million by 2020 (Wong, Kong, and Hui, 2017) and revenues from 
the VR industry are projected to reach more than US$ 40 billion by 2020. There is certainly a rise in the use and 
acceptance of VR technology, especially with the increasing investment in these technologies from major 
technology corporations like Facebook, Microsoft, Sony, HTC and Google, among others. It is apparent that VR 
technologies will continue to be used for learning and training. However, it is important that as these technologies 
become ubiquitous, VR content creators remain cognizant of the significant impact of culture on VR technologies 
and mitigate against these effects on the users. 
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Abstract 


The objective of this study was to evaluate students’ learning-related perceptions and beliefs from an active 
learning case-based instructional strategy implemented in a pharmacy professional course. Weekly active learning 
modules were scheduled ina SCALE-up classroom. Perception data collected with an end-of-semester survey were 
integrated in a regression model. The results of this study confirmed that active learning tasks enhance students’ 
engagement and self-efficacy which, in turn, increases their trust in the effectiveness of their own learning. 


Motivation and Objective of the Study 


Pharmacy students are expected to assume diverse roles once in their profession, including leadership, 
informatics, patient advocacy, and policy development. In order to function effectively in roles that extend beyond 
the traditional roles of dispensing medications and medication management for patients, students need to experience 
instructional tasks that prepare them for this level of content and role diversity. In addition, the 2016 Accreditation 
Council for Pharmacy Education (ACPE) Accreditation Standard 3 emphasizes the need for competence in problem- 
solving, communication, and interprofessional collaboration. 

Persuaded by the importance of active learning strategies to achieve this type of outcome (e.g. Bakon, 
Craft, Christensen, & Wirihana, 2016; Fitzsimons, 2014; Miller, Khalil, Iskaros, & Wan Amburgh, 2017; Pajares, 
1996; Patrio Chiu & Cheng, 2017), the instructor in a Public Health for Pharmacy course redesigned the course 
from a 3-hour weekly lecture to include two parts: a traditional 1-hour weekly lecture and a 2-hour weekly session 
of active learning. The instructional context for the active learning part of the course was a technology-rich 
classroom built based on the SCALE-up model (http://scaleup.ncsu.edu/). Considering the potential impact of this 
active learning setting on students’ learning (e.g. Brooks, 2011), the major objective of this study was to evaluate 
students’ learning-related perceptions and beliefs related to the active learning tasks implemented in the Public 
Health for Pharmacy course. 


Instructional Intervention 


Public Health for Pharmacists (PHRM 540) is a course offered to third-year Doctor of Pharmacy (PharmD) 
professional students. The majority of the PharmD coursework is focused on mastery, retention and demonstration 
of the mastery and recall of facts on exams. Laboratory courses add the hands-on skill into the curricular mix but 
their focus is mainly on demonstration of skills related to the role of dispensing medications in pharmacy. In this 
context PHRM 540 stretches students’ learning habits beyond their comfort zone, because it requires them to master 
new content areas including biostatistics, epidemiology, behavioral science, environmental health, and health policy, 
and to incorporate them into the practice of pharmacy (Strand, Miller, & Focken, 2016). It also requires them to 
analyze new types of information, generate new knowledge, and use the conclusions drawn to solve complex public 
health problems. These course-specific requirements indicate a diversity of content and method compared to more 
clinical-based courses these students are engaged in throughout their academic life. 

In order to increase student engagement with the material and methods in the course, the instructor 
developed and implemented a case-based team-driven active learning strategy in the course. A traditional lecture- 
driven method is used for a traditional 1-hour a week classroom instruction early in the week. Active learning 
modules were scheduled for 2-hours a week in a SCALE-up technology-rich classroom. The case studies used in 
this part of the course required students to apply and integrate the knowledge base they built in the 1-hour lecture to 
solve problems presented in the cases. 
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Students were placed in groups of nine, and seated around tables equipped with surround content display 
technology. Prior to each class, students were expected to individually complete 1/3 of the case. During the active 
learning classroom activity, they started by comparing their work on the first portion. This allowed students to hear 
how other members of their group solved the assigned problems, and to compare answers. To ensure the coherence 
of the group activity during this first part of the activity one person in each group assumed the role of the group 
leader, being responsible at the end of the day for submitting the group’s completed work. Next, students were 
instructed to move on to the second part of the case, and collaboratively complete the case. Most cases had three 
major sections to complete. Several 10-15 minute mini-lectures were inserted by the instructor at critical times with 
the goal to correct misconceptions, emphasise the critical knowledge required for that stage in the case analysis, and 
explain his own rationale for the acceptable solutions for the case. Students were occasionally asked to report to the 
entire class how they used the software to perform a calculation or create an image, utilizing the consoles at their 
own table to project their monitor to the entire class, and speaking over a microphone to ensure audibility to all in 
the room. Three critical competencies in the PharmD program are leadership, informatics, and patient advocacy. To 
illustrate the process of helping students increase and diversify their competence in these areas in preparation for 
being a more well-rounded pharmacist, a few examples will be given. 

First, leadership and communication are skills that need to be developed through practice, and not as 
theoretical concepts. In this course, a different student served as table leader each week with the task to facilitate 
conversation, and to assume responsibility for submitting the table’s group work. Students took this resonsibility 
seriously, and if they did not get full points on their work, the table leader who submitted it was frequently checking 
with the teacher about what was wrong with their work. This proved to be a good practice for helping students build 
the responsibility that goes along with leadership, and allowed them to practice skills of leadership and 
communication along the way. 

Second, informatics, or the ability to retrieve, generate, analyze and interpret data, is an important skill for 
pharmacists. In the cases used in this course, student were frequently given a large data set, which they then needed 
to analyzed, and generate conclusions or recommendations based on the results of their own analysis. For example, 
students were given data on volume of opioid prescriptions nationwide over time, and the number of heroin 
overdose deaths, to look for an association between the two variables. Just telling them that an association between 
the two exists could generate defensiveness among some students, because they could assume that the pharmacy is 
being blamed for the heroin overdose deaths. But when the students analyzed the dataset themselves, and generated 
their own conclusions, they need no persuading for accepting this association. Students beaome deeply engaged in 
trying to explain why that association happen, rather then being defensive about its potential negative stigma to 
phamacies and pharmacists. This exercise required them to analyze new types of information, to generate new 
knowledge, and then to solve problems using actual public health data. This activity was engaging, increased 
students’ self-efficacy, and increases their confidence that they were learning and built the skills to do what was 
expected of them. 

Third, patient advocacy, and the importance of policy that is in the best interests of the patient’s needs, are 
important skills for pharmacists, but these are skills that students do not perceive as interesting or important. As an 
example, by asking students to engage with medication-based problems that they were interested in, such as the 
release of the new and very expensive anti-Hepatitis C drug Harvoni, it was possible to guide the students toward 
considering why they should advocate for their patients, the impact of policy on their patients, and how to address 
problems in these areas. In the Harvoni case study students were first required to analyze data regarding the 
prevalence of Hepatitis C, and the use of Harvoni to treat it. But then the activity moved to a different level required 
student to consider current policies around required prior authorization for patients to be reimbursed for a drug that 
costs $85,000 per patient. They were required to analyze new types of information through consideration of the cost 
of treating all patients in need. They built new skills by determining that the best long-term solution to prevent the 
spread of Hepatitis C is to treat all patients, which then challenges them to consider how this would be paid for. 
Finally, they engaged with reviewing the criteria patients need to meet in order to be granted prior authorization, 
realizing that some of the requirements are discriminatory. This process is engaging for the students, they feel 
competent to make recommendations toward a fair way to manage Hepatitis C patients, and they develop passion for 
justice to address the problem. This happens through the process of evaluating data together as a group, and then 
being challlenged to solve big problems using what they have generated. 

These are just three examples of how this course used engaging activities to increase student interest. 
Increased engagement then contributed to a sense of self-efficacy and confidence that they can perform the tasks 
required of them. Finally, engagement and self-efficacy increased students’ perception of the impact of these 
processes on their learning, which is a proxy for student success. 
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Research Questions & Methodology 


The exploratory question for this study was if students perceived that the active learning section of the course had a 
significant positive impact on their own learning process. 


Research Design 
An exploratory quantitative design research was used to analyze if perceived engagement and self-efficacy 


were significant predictors of students’ perceived impact of the active learning tasks on their own learning. The 
proposed conceptual model is presented in Figure 1. 


Perceived 
engagement 


Perceived impact 
on own learning 


® 


Self-efficacy 


Figure 1. Proposed conceptual model related to the implemented instructional intervention 


As shown in Figure 1, due to the engaging nature of active learning tasks integrated in the target course, the 
expectation is that the active learning tasks increase perceived student engagement and self-efficacy beliefs which in 
turn will increase the perceived impact on their own learning process, a perceptual proxy for students’ performance 
in the course. 


Participants 


The course used for this study had an enrollment of 86 students in their third year of the Doctor of 
Pharmacy program, the last coursework year in the program. A convenience sampling strategy was used, all students 
in the course being invited to participate in the study. A number of 84 (98%) students volunteered to participate in 
this study. 


Data Collection 


The instructor collaborated with an instructional designer to administer online, using Qualtrics®, an end-of- 
course survey with questions specific for three constructs adapted from the literature. The first construct, perceived 
engagement, had six items selected from Gebre, Saroyan, and Bracewell (2014) and adapted to reflect the structure 
and nature of active learning tasks implemented in the analyzed course (see Appendix 1). The second construct was 
a measure of course self-efficacy (Cernusca & Price, 2013) and the third one measured perceived impact of the 
active learning strategies on own learning (Grasman & Cernusca, 2015). The last two constructs were implemented 
with minor changes (e.g. including the course name and building name) to customize them for this study. All three 
constructs used a 5-point Likert evaluation scale ranging from | for Strongly Disagree to 5 for Strongly Agree. 
Scores for each construct were computes as the average of the scores of its individual questions, resulting in a 
continuous score ranging from | to 5. The online survey was administered during the last two weeks of the course 
and students had 10 days to complete the survey. The instructor gave 3 bonus points for those that participated in 
this research study but an alternative equivalent task to earn these points was offered to those that decided against 
participating in this study. This study was approved by the local Institutional Review Board and the informed 
consent form was posted at the onset of the survey indicating the voluntary participation in the study and the 
alternative task available for those interested in earning the bonus points but not participate in this study. 
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Data Analysis 


Data collected were analyzed for outliers, normality and basic statistics using SPSS v13. Analysis of raw 
data revealed two outliers, one for the engagement variable and one for the impact variable. Both outliers were 
deleted leaving 82 participants for the final analysis. Analysis of z-scores (< +/-2.5), skewness and kurtosis (< +/- 
0.5) indicated an accepted level for the normality of the dataset. All three constructs adapted from the literature 
showed a very strong internal reliability with Cronbach’s Alpha values of 0.93 for perceived engagement and 0.92 
for both self-efficacy and perceived impact on own learning. Multiple linear regression was used to test the 
inferential model described in Figure | above. The independent continuous variables were perceived engagement 
and self-efficacy while perceived impact of active learning tasks on own learning was the dependent continuous 
variable. All continuous variables had values ranging from | for low levels to 5 for high level of measured value. 


Findings 


Table | presents the basic statistics for each of the measured continuous variable at the exit point in the 
course. 


Table 1 
Means, Standard Deviations, and Pearson Correlations for continuous variables (N=82) 
M SD 1 2 3 
1. Perceived engagement 4.00 69 - 50** 53 ** 
2. Self-efficacy 3.68 65 - .62** 
3. Perceived impact on own learning 3.55 77 - 


Note: **p < 0.01 (2-tailed) 


As shown in Table 1, all variables had means above the middle point of the evaluation scale (3- neutral) with perceived 
engagement having the highest mean value, which is the main focus of active learning strategies. All correlations 
among the three variables were statistically significant and had medium to high strengths (see Table 1), with the 
correlation between self-efficacy and perceived impact of active learning strategies on own learning being the closest 


one to 0.7, the accepted level for a strong correlation. 


The multiple linear regression calculated to predict the perceived impact of active learning tasks on own 
learning based on perceived engagement and self-efficacy found a significant regression equation, F (2, 79) = 32.83, 
p < .001. The two independent variables, perceived engagement and self-efficacy, explain 45% of the variance of the 
perceived impact on own learning, the dependent variable used in this study (R* = .45). The full results of this 


regression analysis are presented in Table 2. 


Table 2 
Regression Analysis Results (DV-Perceived Impact of Active Learning Tasks on Own Learning Process) 
B SEB B t p 
1. Perceived engagement 33 ll 30 3.09 .0030 
2. Self-efficacy 56 ll 48 4.95 .0001 
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Figure 2 synthesizes the regression findings for the proposed model. 


perceived 
engagement 


perceived impact 
on own learning 


self-efficacy 


Notes: *p < 0.01; **p < 0.001 


Figure 2. Resulted conceptual model tested for the target instructional intervention 

As independent variables, perceived engagement and perceived self-efficacy are strongly associated with 
perceived impact on learning, with self-efficacy having the strongest association. Self-efficacy is a proxy for self- 
perceived competence, and the ultimate goal of the exercises is competence. Engagement is a learning strategy that is 
meant to increase student performance. Therefore, it is to be expected that self-efficacy would be a more sensitive 
predictor of impact on learning than engagement. The goal is learning and mastery, and this appears to have been 
achieved in part by increasing student self-efficacy. 


Discussions and Further Research 


Traditional lecture-driven teaching methods are appropriate for some portions of the PharmD curriculum. 
However, with a format of instruction that included only lecture followed by examination it is very likely to 
underestimate student interest and ability. This approach is frequently a result of assuming that the students are a 
blank canvas upon whom information and knowledge is to be written, and they are dependent upon that transfer of 
information from the teacher to be successful. On the other hand, it is also assumed by some instructors that students 
need to be taught everything they will be asked to perform in laboratory or case-based exercises prior to the activity, 
so they are fully prepared to be successful in it. 

The method described here builds on a middle road by assuming that students have varying levels of 
knowledge and expertise, a gap that is initially reduced by their exposure to an expert-lead presentation of specific 
topics, upon which they can draw when engaging in case study instructional tasks to fully build the expected 
knowledge and skills. Furthermore, by working in groups, the group altogether will be well prepared to approach 
challenges which they have not previously seen, and for which they may not begin with full knowledge. But this 
creates a challenge to the students, and thus contributes to increased engagement, because they feel challenged, and 
not just that they are regurgitating information, or repeating skills they have already been taught. 

This study confirmed that active learning tasks stimulate both students’ engagement with classroom tasks 
and their self-efficacy which, in turn, increases the trust in the effectiveness of their own learning. This strategy 
created opportunities for students to undertake the joint activities of simultaneously increasing their competence in 
informatics, data analysis, communication, collaboration, problem-solving, and development of policy 
recommendations. This allowed for continual reflection on one’s own mastery, and increasing trust in their ability to 
use what they had learned. Previously students had learned most of these skills in the theoretical, and not in the 
practical domain of collaborative work as team members around a table. In addition, this course redesign experience 
indicates that pharmacy instructors can find ways to effectively balance the more traditional lecturing with active 
learning strategies to engage students in more complex and diverse, real-life bound learning contexts while 
enhancing their overall learning experience. 

Considering the piloting nature of this study, the research team intends to further explore the identified 
trends by increasing the depth of the analysis to a more granular level linked to specific active learning tasks 
students perform in this course during the active learning activities. 
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Appendix 1 — Instructional engagement construct 


Compared to my other lecture-based courses, during this courses’ activities Strongly Strongly 
held in the STEM building I... Disagree Agree 

...was engaged more often in discussions with other students on the same table 1 2 Sr AAS 

...had more opportunities to think aloud (expression of ideas, answersto 1 2 3. 4 5 

instructor questions, working on case studies) 

...had more opportunities to engage in reflection on course topic 1 2 364-55 

.. had cooperated more often with colleagues on classroom assignments 1 2 3 4 5 

... was engaged more often in increasing knowledge on course topic 1 2 ay eS) 

...had more opportunities to engage in reflection on my learning 1 2 3.4 5 
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Abstract 


ThinkSpace, a cognitive tool with the underlying premise that people learn by creating explanations for any 
given system, was implemented in redefining an asynchronous online Greek and Roman Mythology undergraduate 
course. This paper elucidates the re-design of interpretive writing assessments as multi-step cases, the affordances of 
the tool, and students’ perceived usefulness of using ThinkSpace in learning the course content. 


Introduction 


One of the critical aspects of learning is problem solving. According to Gagne “the central point of 
education is to teach people to think, to use their rational powers, to become better problem solvers” (Gagne, 1980). 
One way of achieving such learning is by the use of cognitive tools that promote active-learning or learning-by- 
doing (Kozma,1987; Ebner & Holzinger, 2007). Thinkspace is one such cognitive tool that facilitates a scaffolded 
process of interpreting large amount of information based on comprehension of underlying concepts and rules. It 
was initially called the Diagnostic Pathfinder and was built in 2007 by researchers from a large land-grant Mid- 
Western University. It was implemented to teach diagnostic problem solving to veterinarian students. It has since 
evolved with additional tools and is widely used across disciplines such as Communication, Engineering, Geology, 
Classical Studies and others (Bender & Danielson, 2011; Danielson, Mills, Vermeer, & Bender, 2008; Danielson 
1999). 


This paper describes the use of, Thinkspace, to redesign the interpretive writing assignments in an eight-week 
asynchronous online course in Greek and Roman Mythology, that was first offered in Summer 2016 and thereafter 
in Summer 2017. It also shares the student perceptions of usefulness of using such a tool in the course along with the 
instructor perspectives on Thinkspace and the design and development of the course. 


The development of the course was funded by a grant from the online learning unit, within the Colleges of 
Engineering (COE) and Liberal Arts and Sciences (LAS) at a large Midwestern university. The instructor 
collaborated with an instructional designer (ID) as part of the grant. The main role of the ID was to provide 
pedagogical, technical, and instructional assistance. The development process followed an adapted ADDIE model in 
combination with Backward Design (Grant and Wiggins, 2005). 


Course Background 


This Greek and Roman mythology undergraduate course introduces learners to various myths and theories 
of the classical world, gods and goddesses and their relation to religious, psychological, social practices. Courses on 
mythology in general, and on Greek and Roman mythology specifically, are common. Though such a wide 
distribution naturally produces great diversity in specific content covered and the approaches taken to that content, a 
glance at the most commonly used mythology textbooks (Morford-Lenardon, Powell) suggests that the organization 
of such courses is consistently content-based and content-driven. The various major stories of Greek and Roman 
mythology are treated as a body of “fact,” which students are expected to master. That mastery is then demonstrated 
by means of objective assessments - multiple choice and short answer questions on events and characters in the story 
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- with some more subjective assessments mixed in to touch on overarching themes. As mentioned, this orientation 
is reflected in the organization of the major textbooks, which consists of chapters focused on particular characters 
and/or stories. In each offering, though the specific sources and sequencing of material varied considerably, the 
essential philosophy was one of content delivery, with content understood as the characters and events of the 
mythology. 


Historically, this course has always been taught in a face-to-face format. In his first first-to-face offerings 
of the course, the instructor adopted a similar approach. The course was divided into three units, each with a 
different thematic focus: creation myths, myths about gods, myths about heroes. Besides this content focus, 
however, he also adopted as a secondary focus and organizational principle, wherein, modern theories for the 
interpretation of myth with one theory at a time, was introduced in each unit of the course. This approach was 
intended to give the course a “skills based” approach, in keeping with current pedagogical best practices. Through a 
series of face-to-face offerings, emphasis on this feature of the course grew, to the point where a final unit was 
added, focused entirely on myth interpretation, and culminating in a final project in which students were asked to 
choose a method of interpretation to apply to a single myth. 


The learner base in the course consisted of undergraduates from across disciples such as communication, 
political science and engineering. The outcomes of the course were (1) (Understand) Illustrate the major Greek and 
Roman myths; (Understand) Interpret myths in their ancient historical and social contexts; (Analysis)Examine how 
ancient myths pervade the belief structures of their own culture; (Understand)Identify various approaches to the 
interpretation of myth (e.g., psychoanalytic, feminist, structuralist, etc.); (Apply) Apply various theoretical 
approaches to Greek and Roman myth; (Apply) Identify mythological themes in various forms of culture from the 
Renaissance through today. When adapting the course for asynchronous online instruction, the change in format 
combined with the shorter time-table (from a 15-week to an 8-week term) meant the amount of work that students 
could be asked to do in the course - whether reading, writing, or other sorts of activities - would have to be 
increased. In stark terms, there was a choice to be made between breadth and depth. The instructor chose depth, 
and with it a radical change to the structure of the course. Now, skills in the form of myth interpretation would 
provide the backbone for the structure of the course. To make this possible, however, would require deep thought 
and the deployment of carefully developed tools for students to learn the basics of myth interpretation. This is where 
a cognitive tool such as ThinkSpace came in. With the new learning format, it was also necessary to revisit the 
interaction, course structure and the assessments to suit the needs of the online learners as well as be compliment the 
asynchronous online learning platform. The assessments in the asynchronous online format of the course consisted 
of weekly interpretive writing assignments, online discussions, short quizzes and a final paper. 


Design Challenges and Considerations 


There were various factors that guided the design process as the course transitioned to an asynchronous 
online platform. Scalable Design-the online format was chosen specifically to enable a larger enrollment of more 
than 100 students. Class Management-Grading challenges for a high-enrollment class that traditionally has had 
multiple essay-type assessments. Grading such assignments for a large enrollment class could pose a potential 
challenge for the instructor. Hence, such assessments needed to be redesigned. Concept Grasp-Based on the in- 
class experience and feedback, the myths and theories were often challenging to interpret and apply, for learners, 
given their abstract content and unfamiliar context. Providing appropriate avenues to not only grasp the concepts but 
also develop the skills to apply those in a given scenario, in an online format, amplified the challenge. Assessments- 
Students needed to take away not only a knowledge of the facts of mythology but also a practical understanding of 
how to interpret mythology which essentially is a problem-solving skill set. The following section elucidates the 
process by which the interpretive writing assignments were redesigned in Thinkspace and the affordances of the 
tool. 


Thinkspace-The Cognitive Tool 


Cognitive theorists have stated that there is limitation to how much information the mind can process. 
Based on cognitive load theory (CLT) (Sweller, 1988), theoretically, there are three typed of cognitive load-Jntrinsic 
cognitive load (inherently irreducible components of a task); extraneous cognitive load that comes from irrelevant 
tasks under taken to accomplish the task and germane cognitive load that channels the resources in a helpful 
direction to accomplish the task (Pass et.al, 2003). Thinkspace helps in reducing extraneous load and increasing the 


31 


germane cognitive load. In this course, a Thinkspace case consisted essentially of three learning interactions (1) 
Data Collection (2) Data Synthesis- with explanations and appropriate interpretation. (3) Reflection- on the solution 
and comparison with an expert interpretation. In the first two learning interaction, students identified the relevant 
data and then provided with their explanation. In the last learning interacting, students compared their explanation 
with an expert solution. Hence, they receive relevant feedback at the highest moment of engagement for them. This 
increases the germane cognitive load and reduces the extraneous load. 


Building of Thinkspace cases: The above-mentioned learning interactions were translated into a case 
consisting of four phases (Figure 1). In the first phase, students answered guided questions related to the given 
theory and gathered the necessary information (Figure 2). In the following phase, students compared and contrasted 
their line of thought with the ‘expert answers’. In the final step, students wrote their own interpretation of the Myth 
and again self-reflected by comparing it with an ‘expert interpretation’ (Figure 3). All phases were mandatory for a 
case to be deemed complete. Students completed two Thinkspace cases weekly -the first an ungraded practice case 
and the second a graded case. 


Overall, students were completing eight cases that dealt with four different theories for myth interpretation 
and application with varied degree of complexity. Instructor reviewed each phase and provided feedback wherever 
necessary. Thinkspace provided the Instructor the provision to create a comment library over a period of time. This 
proved to be an efficient class-management tool which was used considerably by the instructor for providing 
comments and feedback to the students. Students could also view their responses at any time during the semester. 
Part of the cases included questions related to the final project that students were asked to answer. These responses 
helped students prepare for the final project where they had to interpret a given Myth using the theories they had 
learned during the semester. Both the instructor and students could also contact the technical support any time they 
had any such issues. The response time for such queries was less than 24 hours. 


Hence, the scaffolded learning opportunity, timely feedback, scalability of the assessments, efficient grade 
management strategies and prompt technology support made Thinkspace an appropriate learning tool in this course. 


Phase List 
A list of the total phases that make up this case. 
1. Custom or practice 
Due May 19, 2017 at 11:30 pm 
= 2. Expert review and self-reflection 
Due May 19, 2017 at 11:30 pm 
a 3. Final explanation 
Due May 19, 2017 at 11:30 pm 


a 4. Expert Speaks 


Due May 19, 2017 at 11:30 pm 
Figure 1: The four-different phase in a Thinkspace case. 


Custom or practice 


Assigned reading: HOMERIC HYMNS 1-3. 


You will sequentially interpret this passage by following the steps below. 


Step 1: After reading the story, identify a custom or practice created as a result of this story. 


Step 2: Cite a passage in which the custom or practice is specifically described. 


For this phase, simply copy and paste the passage, from the link provided above. 


Step 3: Identify any god or gods who authorize the custom or practice. 


Figure 2: Question students have to answer before proceeding to the next phase. 
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Expert review and self-reflection 
& 
Expert Review and Self-reflection 


Custom 


Expert Response: 
The mysteries at Eleusis 


Passage: 


“And rich-crowned Demeter did not refuse but straightway made fruit to spring up from the rich lands, so that the whol 
driver, and to doughty Eumolpus and Celeus, leader of the people, she showed the conduct of her rites and taught the: 
into or utter, for deep awe of the gods checks the voice. Happy is he among men upon earth who has seen these myste 


Expert Response: 
darkness and gloom.” 


Gods: 


Expert Response: 
Demeter 


Compare your response with the expert review and write a short reflection below explaining differences, if any. 


Enter your response below: 


Figure 3: Expert Review and Self-reflection 
Students Experience 


This course was an eight-week long Summer course offered in Summer of 2017. Students(N=69) in the 
course were administered two surveys in the fifth week of the course. The ‘Mid-Semester Survey’ which evaluated 
user experience and course alignments and the ‘Thinkspace Experience’ survey to evaluate student perception about 
perceived usefulness of using this tool in the course. The response rate was 69% (N=32). The latter survey was 
adopted from the Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology 
(Davis, 1989). The surveys had a 5-point Likert Scale range ranging from Strongly Agree to Strongly Disagree. 
Both surveys had an open-ended question. The survey was analyzed through descriptive analysis. 


Questions Strongly | Disagree | Neither Agree Strongly 
Disagree | (%) Agree or (%) Agree 
(%) Disagree (%) 

(%) 

The course content would be difficult to learn 3 35 28 31 3 

without the use of the 

Thinkspace. 

Using the Thinkspace gives me greater control over | 6 13 31 44 6 

my learning. 

Using Thinkspace improves my learning of the 6 9 13 63 9 

content. 

Thinkspace addresses my learning needs 9 6 38 41 

Thinkspace saves time in gathering resources 6 16 19 40 19 

Thinkspace enables me to accomplish tasks more 9 22 22 44 3 

quickly 

Thinkspace supports critical aspects of my learning 3 6 13 69 9 

the content 

Using the Thinkspace enables me to accomplish 0 19 22 50 9 

more work that otherwise be possible. 

Using Thinkspace reduces the time I spend on 6 19 31 41 6 

unproductive a 

ctivites. 
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Using the Thinkspace promotes effective learning. 3 9 22 60 6 
Using Thinkspace improves the quality of the work I | 6 16 16 53 9 
produce 

Thinkspace increases my productivity. 3 13 45 32 7 
Thinkspace makes it easier for me to produce my 6 16 16 53 9 
work 

Overall, I find Thinkspace 6 10 16 61 7 
an useful aid in learning the content of the course. 


Table 1: Thinkspace Experience Survey 


Questions Strongly | Disagree | Neither Agree | Strongly 
Disagree | (%) Agree or | (%) Agree 
(%) Disagree (%) 
(%) 
Learning objectives (what I will learn in this course) and | 0 0 0 52 48 


learning outcomes (what I will be able to do as a result of 
this course) are clearly stated and understandable. 


The activities (discussions, group work, homework, 0 0 4 48 48 
presentations) supporty learning in 
this course. 


The instructional materials (video lectures, readings, 4 0 8 46 42 
homework etc.) are current and 
have sufficient depth for me to learn the subject. 


The instructional materials are easy to access and 0 0 4 33 63 
download. 
The assessments (homework, quizzes, exams, projects, 0 0 4 44 52 


etc.) are appropriately timed, scheduled, and appropriate 
to the content being assessed. 


How easy was it to locate the course syllabus and 0 0 8 4 88 
schedule. 
Table 2: Course Navigation Survey 


Overall, the course received a positive feedback from the students as per the Mid-semester survey. 46% of 
respondents agreed that the instructional material had sufficient depth to learn the subject and 48% of respondents 
agreed that the activities in the course supported their learning (Table 2). The Mid-Semester survey results also 
indicated that the learners found it easy to download all instructional materials and that all assessments were 
appropriate to the content being taught. 


From the Thinkspace Experience survey, 63% of the respondents said that using Thinkspace improved their 
learning where as 69% reported that Thinkspace supported critical aspects of learning the content. More than 50% of 
respondents agreed that Thinkspace improved the quality of the work being produced (See Table 1) and that it 
promotes effective learning. More than 60% of respondents indicated that Thinkspace aided their learning the 
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content of the course. The open-ended comments were categorized into the following categories (a) general 
comment (b) learning curve/ease of use (c) aid understanding and application of concepts. Some of the comments 
from the students related to the use of Thinkspace in the course were: 


“T think the learning curve was average, it made you analyze but some of the questions could of been 
better. The expert answers were great. I liked this format.” 


“Sometimes they help and sometimes they don't. It just honestly depends on the format of thinkspace that 
we're doing that week.” 


“T think that having the expert response on Thinkspace really helped me. It was hard to come up with the 
answers 

to those questions on my own but having the expert response helped me to learn where I was right and 
where I was wrong”. 


Overall, the results of the Thinkspace Experince survey and the Mid-Semester Survey indicated that the 
course design had a positive impact on the students. The results indicated that students perceived the tool to be 
useful in aiding them learn the content of the course. The user experience also appeared to be favorable as most of 
the students were able to navigate through the course and were able find the resources and learning content easily. 


Conclusions 


Instructor perspective: I have been generally pleased with the way Think Space has helped my students 
master the difficult material that is applied myth interpretation. There were some early technical difficulties related 
to giving and receiving instructor comments (an essential part of the learning process), but these have been resolved 
in subsequent course offerings. Similarly, the specific steps in each case have been refined more than once, based 
on student results and interactions in face-to-face classes. 


The use of Thinkspace was a novel attempt at re-interpreting the course, specifically the interpretive 
writing assignments. Thinkspace integration scaffolded the learning- breaking down the processes of interpretation 
of a Myth into steps that could be easily adopted by the mind. Secondly, it provided the students an environment 
where their thought processes could be organized and provided ample opportunities of self-introspection and 
reflection before a plausible conclusion. The ‘expert views’ which furthered the opportunities of self-learning and 
reflection (Collin, Brown and Newman, 1989). This approach of problem solving and learning helped in sorting 
through extraneous information and maximized the germane cognitive load (Pass et al., 2003). Furthermore, the 
affordances of the tool, such as expert response, timely feedback, reduced complex grading, a comment library, 
proved to be effective class management strategies for a novice online instructor transitioning into an online format 
in a high enrollment course. Implementing a cognitive tool in this course not only redefined the role of the instructor 
but made learning more engaging for a diverse group of learners. 
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Abstract 


This multi-case study describes the unique experiences of online instructors within a novel context of grant- 
funded online course design assistance at a large Midwestern University. Instructors reflect on their experience, 
from conception to implementation of their online courses, in debriefs conducted by the instructional designers. 
Such reflections can potentially help both, the instructors and the instructional designers, to better understand the 
design process leading to more effective online course designing and implementation experiences. 


Context 


The online learning unit housed within the Colleges of Engineering (COE) and Liberal Arts and Sciences 
(LAS) at a large land-grant research university awards grants to instructors to design and develop their asynchronous 
online and/or blended courses. Grant-recipient instructors, have the opportunity to collaborate with a highly skilled 
instructional designer (ID) throughout the design and development process, where in, the ID provides technical, 
pedagogical and instructional assistance. The online course design process follows an adapted ADDIE model in 
combination with Backward Design (Grant and Wiggins, 2005). At the conclusion of the design phase, and after the 
first implementation of the course, the IDs conduct debrief with the course instructor(s). Such debrief give the 
instructors the opportunity to reflect on their experience designing, developing, and implementing their 
asynchronous online/blended courses. This paper describes the findings of such debriefs and the insights into the 
experiences of the instructors. 


Conceptual framework 


Online learning has revolutionized educational practices with new paradigms, pedagogies, and technologies 
employed to design courses to enhance student learning. Online education provides institutions of higher education a 
low cost, flexible way to reach out to a global audience (Casey, 2008). With the advent and growth of online 
learning platforms in higher education, it becomes critical to understand the role of the online teacher in successful 
online courses and to design courses that enhance student learning. The foremost step in this direction is to 
understand and examine the experiences of online instructors as they design, develop and teach their online 
courses. The Transformative Learning Theory (Mezirow, 1996) can provide the conceptual framework in 
understanding such experiences. In Transformative Learning Theory, “learning is understood as the process of using 
a prior interpretation to construe a new or revised interpretation of the meaning of one’s experience in order to guide 
future action” (Mezirow, 1996, p. 162). Furthermore, Transformative Learning Theory “involves transforming 
frames of reference through crucial reflection of assumptions, validating contested beliefs through discourse, taking 
action on one’s reflective insight, and critically assessing it” (Mezirow, 1997, p.11). Dewey (1933) was the pioneer 
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in reflective thinking and recognized it as a specialized form of thinking. Dewey’s ideas led to the development of 
Schon’s (1983) ‘reflective practice’ and the identification and distinction of ‘reflection-in-action’ (during the 
experience) and ‘reflection-on-action (after the experience) (Schon, 1988). Through critical reflection, instructors 
can critically examine their own teaching practices, personal beliefs, and the implications and assumptions about 
teaching and learning (Larrivee, 2000). Therefore, to understand the experiences of online instructors it is important 
to examine their critical reflections. 


Research Design 


This study followed a multiple-case study approach and examined the debriefs conducted with three 
different instructors. These debriefs were conducted by the IDs after the first iteration of their online course. These 
cases allowed for comparison of the experiences of instructors in designing and teaching their asynchronous online 
courses in different colleges within the university (Yin, 2009). 


Data Collection and Analysis 


The course instructors selected for the study were from COE and LAS. They were novice online instructors 
with no prior asynchronous online teaching experience. The selected courses were offered in the semesters of Fall 
2015-Spring 2016. The invited instructors were informed of the goals of this study and were requested to provide 
with their written consent. This study followed all the requirements of the university’s human subject protection 
office. Pseudonyms were used to protect the identity of the participants and maintain their anonymity. Table 1 
provides the profile of the instructors interviewed. 


Instructor | Age Rank/College | Experience Asynchr 
Teaching onous 
online 
experie 
nce 
Walter 40-50 | Associate 20 years; 0 


Professor/COE | Experience in 
lecture-capture 
distance learning. 
Aron 50-60 | Professor/COE | 20 years; 0 
Experience in 
lecture-capture 
distance learning. 
Marker 60-70 | Professor/LAS | 40+ years; 0 
Experience in 
lecture-capture 
distance learning. 


Table 1: Profile of the instructors 


Ethnographic style semi-structured debrief interview was conducted with each of these instructors, where 
the four stages of ethnographic interview protocol was followed, namely, Apprehension, Exploration, Cooperation 
and Participation (Spradley, 1979). As the instructors were reflecting on their experiences in designing and teaching 
their online courses, it was necessary not to impose any strict structure that would impede their thought process. At 
the same time, probing or follow-up questions were asked at stages where more clarification or detailed response 
was needed. These probing questions were asked mainly under four categories (1) instructor experience with 
designing, developing and teaching an asynchronous online course 2) strengths and weaknesses of the course (3) 
instructor experience working with an ID and (4) instructor’s reflection on course improvements. Each of these 
interviews took place in the respective instructor’s office and lasted about 45 minutes to an hour. The debrief 
interview was conducted by the ID not originally assigned to work with the instructor during the designing phase of 
the course to avoid any potential conflict or bias in the instructor’s response. 

The audio recordings were transcribed and analyzed individually by the three researchers involved in this 
study. Each researcher read a transcription in its entirety to develop a sense of the data. Subsequently, the research 
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team met to code one transcription together line by line. As they coded, they added key words and phrases that 
capture some initial ideas of the data. 

Thereafter, the researchers met to discuss each of the analyzed transcripts. They discussed the preliminary 
codes and resolved any discrepancies. After all the three transcriptions were revised and open coded, a preliminary 
codebook was developed. The within-case analysis allowed the researchers to become more familiar with the cases. 
Using preliminary code book, the researchers coded the transcripts individually again and met in pairs to review 
those. Through a peer reviewing phase, the researchers reviewed each transcript and updated the codebook. Any 
disagreements were discussed and the researchers resolved them by discussing and reaching a consensus 
(Brinkmann and Kvale, 2015). Adjustments were made, new themes were added, if needed, and a consensus was 
reached for a final code book. Using the final version of the code-book, each of the three transcripts were coded by 
each researcher for the last time and compared, the inter-rater reliability of more than 85% was established and the 
emerging themes were finalized. 


Creating Codebook 


ry 


Reviewing Individual 


i Individual Re-coding + Developing Themes 
Coding 


Individual Coding ——> 
* 
é 
Peer-reviewing Individual 
Re-coding 


Figure 1. Cycle for Data Analysis 
Findings 


Each instructor had a unique journey and by cross-analyzing the ethnographic interviews some common 
categories emerged: (1) teaching philosophy (2) instructor’s learning during design and development process (3) 
online experience during implementation of the course (4) instructor’s perception of online learning benefits (5) 
instructor’s challenges during course designing and during implementation phase (6) support from ID (7) course 
improvements (8) instructor’s content development experience. The frequencies of each code occurring in the 
transcripts was also calculated. After considering the frequencies of each code and a through cross-examining of the 
categories across the cases along with relevant evidence from the transcripts, two main themes emerged (1) 
Evolution of Instructor’s understanding an asynchronous online course: Content planning and Interactions (2) 
Working with an Instructional Designer. 


(1) Evolution of Instructor’s understanding of an asynchronous online course: Content planning and 
Interactions All instructors expressed that they had experience in teaching an online course. However, their 
understanding of an online course was restricted to the lecture capture format. In such a format, a live face-to-face 
class would be recorded and distributed to distance students, who either would log-in to the class synchronously or 
view the recorded materials at a later time. The Learning Management System (LMS) was used primarily used as a 
repository for lecture notes and for the delivery of recorded materials. All the three instructors, included in this 
study, had no prior experience in designing and teaching an online course. Their understanding of such a learning 
format evolved as they worked with an ID during the process of designing and teaching their online course. This 
evolution fell into two overarching areas of content planning and interactions which has been described in the 
following sections. 


“This experience had a rather dramatic change in the way I teach my course’’-Marker 

Content Planning: All instructors acknowledged the initial time-investment in the content planning phase 
to develop their asynchronous online course. This phase included listing of their learning outcomes and objectives, 
modularizing the course, sequencing of content and creating content. They also acknowledged that the time invested 


in such content planning processes was far more than their previous face-to-face teaching experiences. 


“Tt was general a good experience although it took longer time than expected for preparation and trying to 
perfect the lectures and so on”’-Aaron 
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Furthermore, modularizing the whole course into logical modules, creating a detailed course schedule with 
precise due dates for all deliverables, and sequencing of the content were some of the firsts for these instructors. 
These came to the forefront in the debrief interviews. 


“T think the other thing that I did differently this semester was the syllabus and the schedule document that 
you help me provide. It was a lot more detail than anything I'd ever had before. And I think that was very useful’”’- 
Aron 


Asynchronous recording of lectures, planning and preparing for the same, and segmenting these into 
shorter video segments were not only challenging but presented the instructors with a new perspective at looking at 
the content and its planning. 


“so I wasn’t sure about the recordings, it took a while for me to figure out how to record the videos, edit 
videos, it was quite time consuming actually because it is a learning process for myself. I haven’t done it before, 
initially was very time consuming, like it took me one day maybe even more to record just one lecture of maybe 60 
minutes ”.-Marker 


“I mean it took some time to think about how to partition the material, trying to have things in the segment 
that would hang together, and have segments that were roughly of the correct length” — Marker 


“breaking the lecture into smaller parts. This was actually difficult for me to understand the students would 
like to see 15 minutes or 20 minutes maximum lecture.”’-Walter 


Interactions: Understanding interactions between student-student and student-teacher, in an asynchronous 
online course, was a learning experience for the instructors. With their prior face-to-face experience, interactions 
mainly consisted of in-class questions by students. Any course updates would be verbally announced to the class by 
the instructor(s). Instructors relied on verbal and visual cues while delivering their lectures and accordingly paced 
the content covered in class or provided remedial lectures. However, in the asynchronous environment absence of 
such interactions made the instructors revisit their established teaching strategies. 


“T’m in the classroom, I implore my students to ask questions anytime that they want. I tried to stop 
periodically to ask if there are questions, but of course, you can't do that in the asynchronous fashion. ’’-Marker 


“how to pace the lecture, that part is different, in the in-classroom lectures, I could have interactions with 
the students and then I can adjust the pace adaptively, If I know that students are not getting it, then I can stop it and 
slow down, gave more hindsight”-Walter 


Interactions were redefined according to the affordances of the LMS or the online meeting applications, 
such as Zoom. Instructors were initially apprehensive of the type and quality of interaction in the online format. 
Gradually, they appreciated the various channels by which interaction was facilitated in such courses. They also 
valued the frequency and quality of communication. This was a big leap of their evolved understanding of the online 
asynchronous format. Instructors, in their respective courses, had online office hours at scheduled times and days 
where students could come and interact synchronously with the instructor. Other LMS tool such as the Discussion 
forums, both graded and non-graded, was utilized to facilitate interaction. According to the instructors, such 
platforms promoted student-student engagement in their courses. 


“The discussion forums in fact supplemented the completely asynchronous teaching mode by having 
students actually engage in the discussions and so on.”’-Aaron 


“for students to get them engaged, the online students for example, I think the discussion forums were 
actually very good rather than just talk to the computer, the students watching the videos and so on so engaging 


them in the discussion through the discussion forum, this was actually very helpful’’-Aaron 


Any critical information shared in the discussion forums or via emails was compiled by the instructors and 
shared with the class using Announcements. This process of communicating further evolved their understanding of 
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the asynchronous online platform which necessitated them to reflect on the course in an ongoing manner. This also 
helped them maintain a regular stream of communication with the students. 


“Also, they interact among themselves online, and that’s good because if one student has a question then 
others can help him or her.”’-Walter 


“.my policy has always been to take the email questions provide an answer and then with very few 
exceptions strip off all the names and identifying information and send it to the entire class. ”-Marker 


Analyzing the debrief transcripts also brought the working relation between the instructor and the ID to the 
forefront, the second major theme that emerged. 


(2) Workflow with the Instructional Designer: The IDs working with the instructors in this study have 
more than three years of experience in designing asynchronous online courses. They worked with the instructors 
over period of 16-18 weeks consulting, assisting and advising at each step of the design process. Typically, the 
workflow included regular on-going meetings, discussions on course objectives and outcomes, content sequencing, 
assessment design and LMS page design. The IDs also work with the instructor during the implementation phase of 
the courses providing technical, instructional and pedagogical advising and assistance. All the three instructors in 
this study acknowledged the value that the IDs bring to the design process. 


“T could not have put this course together and anything close to its form on without the help that I got 
because I was totally clueless and so you are very helpful and I appreciate the responsiveness and the accuracy of 
all things that I got from you during the semester’-Marker 


Instructors expressed their appreciation for the guidance received in terms of content sequencing, i.e., 
length of each module, pacing the different assignments and type and format of assignments. They also 
acknowledged the guidance related to creating their syllabus, schedule and LMS page design which included course 
structure, layout and navigation. 


“Also you proposed for different methods of assessment, specially, actually discussion forums, in fact, I 
think they played a pivotal role in getting the online students involved because without these discussion forums that 
you have suggested I think that the interaction with online students would have been to a much lesser degree’’-Aron 


The instructors appreciated the regular meetings, detailed minutes and reminders by the ID(s) to keep the 
project moving forward at a timely pace. 


Also, follow up on the deadlines, also the email messages that you used to send us, actually this kind of, put 
some kind of pressure actually, not in a bad way pressure on us to pay more attention to record the lectures and put 
the material online. Otherwise, things may have dragged a little bit.”-Aaron 


“..the responsiveness is excellent and very helpful I never could have done this without the help that I 
got..”’-Walter 


The findings of this multi-case study reveal that conducting debrief help unveil instructors’ inner thoughts, 
struggles, and approaches through a reflective practice. The challenges of content development, be it depth, time or 
technical, has come to the forefront of the design process. Interaction between students and between student and 
instructor was also a critical factor that shaped the experiences of these novice online instructors. Through the 
debrief interviews it became clearer that the instructors considered the design process far more front-load heavy than 
theory previous experiences. They also expressed that this experience reshaped many of their existing beliefs, 
teaching strategies and practices, and pedagogical assumptions in terms of student engagement, assessment design 
and course facilitation. The value that an ID brings to the entire design process also came to the forefront through 
the debriefs. Through debriefs, it has been possible to have a better understanding of the instructor’s individual 
journey and their working relationship with the ID. 
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Conclusions 


The debriefs not only shed light on the instructor’s individual experiences but is also a step towards making 
the them a reflective practitioner. A reflective practioner is seen as a combination of self-awareness, reflection and 
critical thinking (Eby, 2001). Conducting debriefs can be a critical tool for the instructors to reflect on their design 
processes and teaching philosophies. It also serves as an insightful tool for IDs to improve their workflow. A better 
understanding of processes and philosophies, for both the instructor and the IDs, potentially can lead to more 
effective course design collaborative practices. 


Future Study 
This study can be extended to examine how such reflections enhance the design process and lead to better 
asynchronous online courses. Expanding debriefs with experienced asynchronous online instructors can unearth 
newer themes. This study could potentially be further extended to add to the ADDIE model to include debriefs as 
part of the instructional design process. 
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Abstract 


The NC Department of Public Instruction’s Digital Teaching and Learning Division engaged district teams 
across the state in regional professional learning on the Digital Learning Competencies throughout July 2016 and 
implemented the competencies for all school districts in July 2017. The North Carolina Digital Learning Plan 
provided recommendations for state actions that support K-12 schools, as they become digital-age learning 
organizations. The purpose of this study was to explore the perceptions of school administrators about the NC 
Digital Teaching and Learning Competencies for school administrators, for classroom teachers, and to identify best 
practices for assessing faculty. The Principles of Transformational Leadership to develop digital leadership skills, 
and Merrill’s First Principles of Instruction for the design and development of professional learning, guide the 
conceptual framework. Using a mixed method concurrent approach, data was collected during a pilot study fall 
2018. Quantitative data collection occurred through a web-based questionnaire distributed to NC school 
administrators (n=21) in one school district, and qualitative data was collected by individual semistructured 
interviews with a homogeneous group of 6 NC school administrators from one southeastern district. For the 
quantitative data analysis, descriptive results were categorized based on the five focus areas outlined in 
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administrator competencies. Thematic analysis of the transcribed interviews followed an open coding process to 
identify categories and themes related to administrators’ perceptions about teaching and learning competencies and 
best practices assessing teachers about the competencies. The goal from this study will support the design and 
development of a digital learning certificate to support administrators to address the knowledge gap in digital 
leadership practice to support teachers and students. 


Introduction 


The teaching and learning process is a complex balance of content knowledge, pedagogical strategies, and 
technological resources to actively engage and support learners. In 2016, the North Carolina (NC) state board of 
education approved the integration of the Digital Learning Competencies for Teachers (DLCT) (see Appendix A) 
and the Digital Learning Competencies for School Administrators (DLCSA) (see Appendix B) for K-12 schools 
statewide. The DLCT and DLCSA, informed by International Society for Technology in Education (ISTE) 
Standards, International Association for K-12 Online Learning (INACOL), and the NC Professional Teaching 
Standards, are to be viewed within the context of the current North Carolina Professional Teaching Standards as 
extensions in relationship with the ways that digital technologies impact and affect schools. Conversely, the DLCSA 
are to be viewed within the context of the current North Carolina Standards for School Executives as extensions in 
relationship with the modes that digital technologies impact and affect schools. School and district administrators 
should use DLCSA to improve practice, build capacity in the staff, and drive student learning within their schools. 
Each focus area in the DLCSA conveys a subset of competencies that help to explain and ‘unpack’ the five focus 
area: (a) Vision and strategy (b) Content and instruction, (c) Human capacity and culture, and (d) Personal growth 
and connectedness (e) Community. Throughout DLCSA is the underlying assumption of leadership and excellence 
with regard to digital citizenship. Additionally, administrators should model the behavior they expect from their staff 
and students and should continually seek to represent their schools and districts with the way they convey 
themselves both on and offline. Teachers and administrators should use the competencies to improve practice and 
drive student learning within their classrooms. 


Background and Context 


In NC, school and district administrators are asked to follow the DLCSA to improve their practices, support 
schools, teachers, and students in digital learning environments. In 2013, based on initiatives from the state’s 
General Assembly, the State Board of Education developed a set of digital teaching and learning competencies 
providing a framework for “schools of education, school administrators, and classroom teachers on the needed skills 
to provide high-quality, integrated digital teaching and learning” (NC Department of Public Instruction, n.d .). 
Effective in July 2017 for the beginning of the 2017-2018 academic year, the goal of the digital teaching and 
learning competencies is to provide a framework for teachers and school leaders to demonstrate skills essential for 
instructional practice that creates a digital learning environment for students in the digital age. Additionally, the goal 
of the DLCT and DLCSA is to support educators, communities, and administrators transition to digital-age teaching 
and learning and assist school district teams to reflect on current practice while identifying opportunities to support 
digital schools. The DLCT and DLCSA were developed in conjunction with the ISTE NETs-T and NETs-S and 
reflect the changing nature of schools in NC as well as nationwide. Technology has become a conventional tool for 
students to learn as well as becoming a necessity to develop digital-age ready students as they transition from K-12 
schools settings to successful digital citizens in higher education institutions, military, and careers. As technology 
available to schools and students changes, the manner educators approach teaching and learning must also adapt. 
The DLCT and DLCSA provide teachers and administrators with a framework for making those changes. 

The long term goals of the North Carolina Digital Learning Plan includes “Teachers in all classrooms who 
meet the digital learning competencies standards and are able to effectively apply digital-age approaches, tools, and 
resources to support their students’ learning” by the 2019-2020 school year (Friday Institute for Education 
Innovation for the North Carolina State Board of Education -Department of Public Instruction [FIEI], 2015). To 
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accomplish the goal of ensuring all classrooms across the state integrate and adhere to the digital learning plan, 
district administrative leaders should identify the appropriate digital leadership skills and technology knowledge to 
model behaviors that enable their teachers to be successful in the classroom. Primarily, basic infrastructure and 
personnel necessities should be assessed and evaluated to identify current gaps to ensure updates to meet current and 
future requirements. Infrastructure challenges schools in NC identified as a barrier for successful implementation of 
digital learning competencies were outmoded building infrastructure, incompatible equipment, and antiquated 
systems. The capacity to run and maintain digital devices and requisite software has become challenging for schools, 
especially those in rural areas (FIEI, 2015). Maintaining a school district’s technology infrastructure is a complex 
and expensive endeavor impacting daily functions for administrators, faculty, and students. 


Currently, a national emphasis on standards-based accountability based on investing in and implementing 
digital technologies in the classroom exist for administrators and teachers. The purpose of this mixed method study 
was to explore the perceptions of school administrators about the DLCSA and DLCT to identify best practices for 
assessing classroom teachers on the competecies. 

Literature Review 

As technology becomes pervasive in society, K-12 students are connected and dependent on technology 
and digital devices for communication, information, and learning. To support digital learning, statewide school 
initiatives are implementing various digital technology plans across our nation (Sheninger & Murray, 2017). 
Effectively leveraging technology to advance pedagogical practice in the classroom to support learning is a critical 
issue facing administrators. Developing professional digital learning opportunities to support school administrators 
as digital leaders could lead to advocacy for teachers and students. Supporting administrators to embrace digital 
leadership and fully understand the digital learning competencies teachers integrate could lead to improved 
assessment practices. Transformational leadership theory supports administrators’ vision for change, and Merrill’s 
First Principles of Instruction (FPI) provide the conceptual framework to guide the design and development of 
professional learning opportunities to support school administrators as digital leaders (Merrill, 2002). 


Principles of Transformational Leadership 


According to Burns (1978), transformational leadership occurs when one or more individuals engage with 
others in a mutual process where “leaders and followers raise one another to higher levels of motivation and 
morality” (p. 4). Burns (1978) originally introduced the concept of transforming leadership in descriptive research 
on political leaders. The transformational leadership approach supports leaders producing an organizational vision 
beyond existing systems and practices guiding change (Bass & Avolio, 1994). Additionally, Avolio and Bass (1994) 
discovered that transformational leaders: (a) inspire loyalty, (b) encourage followers to express their own ideas and 
opinions, and (c) inspire followers to be proud to be associated with them. Transformational leaders take risks to 
focus on transforming and encouraging change and innovation to support the organization as a whole (Eliophotou- 
Menon, 2016). Fairholm (2001) discussed the benefits of applying the transforming leadership approach to focus on 
the more personal side of organizational interactions. Terms such as “vision, culture, values, development, 
teamwork, and service make sense in the world of transforming leadership” (Fairholm, 2001, p. 2). School 
administrators embracing the transformational leadership approach focus on developing a collaborative, supportive 
educational environment, which would support digital teaching and learning competencies. 

Historically, K-12 principal’s role has been process-based that is concerned with managing daily operations 
of the building, teachers, community partners, and students (Curnyn, 2013). However, within a changing 
educational climate and accountability for digital learning, the principal’s role continually evolves changing focus 
being process-based to outcome-based leadership. As schools shift to create digital-age learning environments 
supporting a resilient digital culture, schools need to develop leaders with a clear vision along with teachers willing 
to follow that leadership style (Curnyn, 2013; Eliophotou-Menon, 2016). Transformational leaders support and 
motivate teachers to accomplish goals and design an organizational environment respectful of individual needs and 
forming a collaborative culture toward the achievement of common goals (Eliophotou-Menon, 2016). While the 


45 


principles of transformation leadership support administrators to lead change in the digital age, Merrill’s FIP provide 
a framework for the design and development of professional learning opportunities to promote successful digital 
leadership modeling. 


Merrill’s First Principles of Instruction 


According to Reigeluth (1999), regardless of program (approach to prescribed practices) or practice 
(instructional activity), a principle defines a relationship that is always true under appropriate conditions. Merrill 
(2002) identified shared themes within instructional design theories and models to determine commonalities among 
terms. While terminology may differ across instructional design theories and models, “principles are necessary for 
effective and efficient instruction” (p. 44). The First Principles of Instruction (FPI) are interrelated principles applied 
to instruction to increase student-learning outcomes. The principles should be used to create learning environments 
instead of explaining how learners create knowledge (Merrill, 2002). Learning is promoted as a result of: (a) solving 
real-world problems, (b) existing knowledge is activated as a foundation for new knowledge; (c) new knowledge is 
demonstrated to the learner; (d) new knowledge is applied by the learner; and new knowledge is integrated into the 
learners practice (Merrill, 2002). The goal of this study will support the design and development of a new digital 
learning certificate to support administrators as digital leaders and to address the knowledge gap in practice 
assessing teachers on the digital learning competencies in the classroom. Merrill’s (2002) FPI suggests that prior 
knowledge can activate opportunities to support new knowledge, which supports this instructional design situation. 


Administrators’ Digital Leadership 


According to Domeny (2017), digital leadership is a new leadership construct for K-12 administrators that 
connect them with technology. Digital leadership encompasses “using the vast reach of technology (especially the 
use of social media) to improve the lives, well-being, and circumstances of others (Couros, 2013, p. 1). Digital 
leadership requires administrators employing a mindset leveraging a strategic view of the school’s culture, which 
focuses on student achievement and how students and teachers engage with technology (Askal, 2015; Sheninger & 
Murray, 2017). For these systemic changes in pedagogical practice and student learning to occur, education leaders 
should create a shared vision for how technology can meet the needs of all learners while developing a realistic plan 
that translates the vision into action. Research indicates that administrators creating a digital culture with teachers 
and within the school’s community that will help support the needs of today's learners (Arokiasamy, Abdullah, & 
Ismail, (2014); Askal, 2015; Domeny, 2017). With society becoming increasingly dependent upon technology it is 
inevitable for leaders to connect the power of digital technologies to create school cultures that are “transparent, 
relevant, meaningful, engaging, and inspiring” (Sheninger & Murray, 2017, para. 1). Encouraging school 
administrators to embrace digital leadership roles requires cultivating new digital technology support and learning 
opportunities. Sheninger and Murray (2017) found that digital leadership flourishes based on the leader’s symbiotic 
connection with technology, and the principal’s support for professional development to support digital technologies 
in the classroom. 

Askal (2015) suggested that school leaders are experiencing a gap in knowledge and application navigating 
how to lead and support educators with newly required digital developments and integrate practices into school 
learning environments. Examining school culture resulting from principals as digital leaders, 93% of the principals 
stated that while they were aware of digital leadership and importance for improving teaching and learning 
outcomes for the digital age, they experienced limited opportunities to implement digital leadership in their school 
due to the time required for improved professional development and training, and lack of technological 
infrastructure, along with financial constraints updating the current technology infrastructure (Askal, 2015). 
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Administrators’ Digital Technology Skills & Practice 


Technology standards for teachers at the state and national level provide a foundation for goal setting and 
classroom integration for teachers to improve instruction in the K-12 classroom (Britten & Cassidy, 2005). “It is 
critical to the successful infusion of technology standards that there be a means by which classroom teachers can 
identify the connections among standards, best practices in teaching, and uses of technology (Britten & Cassidy, 
2005, p. 50). In 2018, the International Society for Technology in Education (ISTE) developed Standards for 
Education Leaders (SEL) to support educational leaders prepare for leadership practice to support digital learning 
strategies in schools: (a) equity and citizenship advocate, (b) visionary planner, (c) empowering leader, (d) systems 
designer, and (e) connected learner (p. 1-2). The first standard, equity and citizenship advocate, described how 
school leaders use technology to increase equity, inclusion, and digital citizenship practices. Education leaders, 
through skilled teachers, technology access, modeling behaviors, critical evaluation, and cultivating responsible and 
digital technology behaviors and environments, demonstrate digital citizenship (SEL, 2018). 

Digital technology produces opportunities for growth and could require a new flexible leadership model for 
effective administrators to support digital leadership for teachers and students. Administrators that model solid 
technology integration skills and support faculty meeting required technology standards in the classroom, help to 
transform learning (Zhong, 2017). Determining administrator’s technology skills strengths and weakness could lead 
to increased technology integration efficacy. Cheung and Slavin (2013) determined that technologies used by 
educational administrators reflect their attitude toward a digital management philosophy. A positive administrator 
digital management philosophy was reflected in teacher technology experimentation and usage (Daraghemah & 
David, 2017). School administrators participating in digital technology related continuing education and 
professional development training to learn and incorporate digital technology skills lead to a positive attitude toward 
overall digital learning within the school (Daraghemah & David, 2017). 

Professional learning opportunities matching the digital leadership vision are vital to the implementation 
process by teachers. Only through ongoing, consistent, leveled professional development opportunities for teachers 
and administrators can users gain the technology skills and confidence needed to implement the new digital 
literacies (Larson, Miller & Ribble, 2009). Larger districts may have instructional technology staff to provide 
necessary ongoing support for teachers and administrators; however, smaller districts may need to identify 
alternative methods to fill the gap in practice. Gaining an understanding about how technology standards directly 
affect members of the school can assist in the identification of best practices for digital leadership by administrators. 


Administrator’s Perception about Technology Leadership 


School administrators directly affect how technology standards are perceived by classroom teachers, 
students, and community partners. Additionally, principals’ modeling strategies directly affect implementation of 
digital learning strategies in the classroom (Fisher & Waller, 2013). According to Fisher and and Waller (2013), 
the school principal’s responsibility in shaping the teachers’ beliefs and attitudes toward a shared vision using high 
quality instruction and technology integration could remove barriers to successful technology implementation in the 
classroom. The absence of administrative support could be the most significant barrier to technology integration for 
teachers (Fisher & Waller, 2013). “Leadership has a major impact on education technology usage, leading to 
improved student outcomes” (Greaves et al., 2012, p. 14). The most effective method for school administrators to 
promote technology integration is to become a knowledge and effective user of technology (Fisher & Waller, 2013). 

Guided by the five ISTE-A Standards for Education Administrators (2009) Metcalf and LaFrance (2013) 
examined principals’ perceptions about technology leadership preparedness, and found that digital citizenship was 
the most prepared indicator while visionary leadership was the least prepared indicator. Additionally, ISTE-A 
standards should align and incorporate higher education principal preparation programs as well as district 
professional development (Metcalf & LaFrance, 2013). Creating and compelling an educational environment from 
the perspective of visionary leaders should influence the integration of digital technologies in the teaching and 
learning process (Curnyn, 2013; Kowalski, 2010). Additionally, visionary leaders should seek and promote 
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communication and collaboration within the building and among community stakeholders. However, Zhong (2017) 
found that 254 public school teachers from five districts in Mississippi did not value visionary leadership strategies 
as an effective practice. Principals are most effective when supporting teacher professional development and digital 
citizenship (Zhong, 2017). Additionally, teacher technology training and support is integral for integration efficacy 
(Zhong, 2017). 

Ribble and Miller (2013) discussed the importance for school administrators to adopt and support a digital 
technology plan to support a digital learning environment allowing teachers to meet technology standards within a 
creative, innovative environment. When administrators gain an understanding about teacher technology standards 
and develop transparent strategies in conjunction with each school’s improvement plan to assess best practices for 
technology integration supporting pedagogical practice in the classroom, open communication and collaborative 
environments emerge. 


Digital Technology Assessment 


Technological changes in the digital age challenge schools to integrate innovative technologies in 
pedagogical practice in the classroom and throughout the K-12 curriculum (Kozma 2010). However, there appears 
to be a debate as to whether digital technology integration improves student learning. After reviewing several studies 
about whether there is a clear connection between information and communications technologies and student 
learning outcomes, it appears as if student-learning outcomes are a far more complicated relationship than simply 
availability of usage (Kozma, 2010). To date, each state in the United States has latitude to support the ISTE 
standards and assess classroom teachers on meeting those standards through each state education association and 
local education association (ISTE, 2018). As school systems across the United States develop strategies to support 
digital technology integration in the K-12 classroom, assessment strategies identifying and evaluating teachers’ 
breadth of knowledge and types of technology utilized should be developed; moving beyond district created rubrics 
that delineate whether standards have been met or not met. 

Traditionally, assessment has been related to how well teachers measure student achievement and student 
learning outcomes (Stiggins, 1991). However, assessing whether K-12 teachers have successfully integrated digital 
technology standards into the classroom carries a great deal of subjectivity. A plethora of assessment models and 
theories have been developed to support teacher assessment, however, to date, research has not described how 
administrators should determine whether teacher technology standards have been met in the classroom. Key research 
does support the importance for administrators to become digital technology users through continued professional 
development training to support K-12 teachers in classroom integration. 


Methods 


Using a mixed method approach (Creswell, 2013), data was collected during a pilot study Fall 2018. A 
non-experimental descriptive design was appropriate for the quantitative data collection for this pilot study study 
because the aim was to examine perceptions of school administrators about the digital learning competencies. 
Nonexperimental research lacks the manipulation of an independent variable, random assignment of participants to 
conditions or orders of conditions, or both (Reio, 2016). Following Internal Review Board approval, a letter of 
cooperation was submitted and approved by a school district superintendent in southeastern North Carolina. 
Following approval from the district superintendent to conduct research, an email explaining the purpose, goals, and 
objectives of the study was sent to potential participants. All district principals and assistant principals received the 
participation email. A link to the web-based questionnaire beginning with informed consent was included at the end 
of the study participation email. 

Quantitative data collection occurred through an electronic questionnaire distributed to NC school 
administrators in one district in a southeastern county (n=21), and qualitative data was collected with individual 
semistructured interviews with a homogeneous group of 6 NC school administrators. For the quantitative data 
analysis, descriptive results were categorized based on the five focus area supporting the DLCSA: (a) Vision and 
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strategy (b) Content and instruction , (c) Human capacity and culture, and (d) Personal growth and connectedness (e) 
Community. Qualitative data analysis included thematic analysis of the transcribed interviews followed an open 
coding process to identify categories and themes related to administrators’ perceptions about teaching and learning 
competencies and best practices assessing teachers about the competencies. Content validity of the measures was 
validated by three experts: researchers in the field of instructional technology and by two school administrators. 
Web-based questionnaire demographic information and results can be located in Appendix C. 


Pilot Study Results 
Web-based Questionnaire 


Twenty-one respondents completed the demographics part of the web-based survey, however, all 
respondents did not answer the remaining questions (n=17/18) regarding the perceptions of school administrators 
about the DLCSA. Table 1 describes the demographic data collected including gender, age, job title, school location 
and characteristic, years of teaching experience, and years of school administration. 

Awareness of the DLCSA. Based on survey results from the web-based questionnaire, 82.36% (n=14) of 
school administrators’ are familiar with the NC DLCSAs directive from the NC Department of Public Instruction. 
The following sections present the descriptive results from the questions related to the perceptions of the five focus 
areas of the DLCSA. 

Vision and strategy focus area. According to the NC DLCSAs competency vision and strategy, school 
administrators will engage to create and communicate a vision for digital teaching and learning in their schools, 
embedding into the strategic plan for implementation and execution. Additionally, administrators should cultivate 
and articulate a clear and relevant vision and strategy for digital learning, advocate for, prioritize, and ensure 
equitable, sustainable access to available technology resources and encourage full participation of all learners in a 
digital learning environment (ISTE, 2018). Additionally, administrators should plan for and use funding effectively 
to support and sustain a vision for digital learning. Facilitate a school improvement planning process that is centered 
around personalized learning supported by digital learning environments. 

Overall, respondents believed that they have somewhat met the standards for vision and strategy for their 
school, staff, and students for digital learning. The majority of respondents perceived that they significantly 
advocate equitable, and sustainable access to technology for staff and students. Almost half of the respondents 
perceived that their school could use funding specifically to support digital technology learning. Additionally, half 
of the respondents responded that they were unable to successfully implement a vision and strategy in their school 
due to a lack of funds or the need for additional training for the staff. More than half of the respondents perceived 
their ability to facilitate a school improvement planning process to support personalized learning for the staff as 
somewhat effective. 

Content and instruction focus area. Content and instruction requires school administrators to be the ‘lead 
learners’ in their schools, model appropriate instructional practices and ensure content that encompasses appropriate 
digital tools, resources, and pedagogies. Promote and model positive digital citizenship as well as implement 
practical policies for communication and collaboration with stakeholders to ensure responsible, effective digital 
teaching and learning practices throughout all school processes. Actively advance and promote digital competencies 
for teachers by increasing access, opportunity, and resources for professional growth and the development or 
acquisition of instructional materials. Establish and use systems to analyze and share data to guide whole-school and 
classroom-level continuous improvement, and establish and use systems for the acquisition, vetting, creation, and 
implementation of digital content as well as evaluation systems for effectiveness. 

Fifty percent of the respondents perceived that they had significantly modeled and/or fully promoted 
positive digital citizenship. Forty-four percent of the respondents responded that they perceived they actively 
advanced and promoted digital competencies for teachers by increasing resources; however, 44% of the respondents 
perceived that they only somewhat promoted digital citizenship. More than 50% of respondents stated that they 
perceived to have fully or significantly established and used systems to analyze and share data with their staff. 
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Additionally, 50% perceived that they somewhat or minimally established and used systems for creation or 
implementation of systems to analyze and share data with their staff. 

Human capacity and culture focus area. Human capacity and culture describes how administrators will 
leverage digital tools and resources to further develop a positive culture of learning that seeks continuous 
improvement among staff and students. In order to meet this competency, administrators must allocate time, 
resources, and access to support digital learning efforts, maximize capabilities of the school staff, and ensure 
ongoing professional growth for self and staff, provide learner-centered environments equipped with appropriate 
learning resources, including digital technologies, to meet the diverse needs of all learners. Additionally, 
administrators will need to develop a plan to build technology, pedagogy, and content knowledge capacity in current 
staff members and create channels for the strategic recruitment of talented new hires. Actively supporting staff 
through effective modeling and coaching practices in conjunction with utilizing relevant digital technologies to 
facilitate reflective two-way feedback is key to supporting human capacity and cultivating a digital learning culture. 

More than half of the respondents perceived that they fully or significantly have allocated time, resources, 
and access to support digital learning efforts in their school. When asked about providing a learner-centered 
environment, more than 50% of the respondents responded that they have fully or significantly a provided learner- 
centered environment. Forty-one percent of respondents stated that they have significantly built technological 
pedagogical knowledge and content knowledge into the current technology plan, and 41% of respondents stated that 
they somewhat included technological pedagogical knowledge and content knowledge into the current technology 
plan. Thirty-five percent of the respondents perceived that they actively support staff by modeling and coaching 
practices; however, about 35% of respondents discussed that they only somewhat support staff by modeling and 
coaching practice. 

Personal growth and connectedness focus area. Personal growth and disconnectedness describes how 
school administrators will develop a personal learning network and demonstrate a dedication for continued growth 
and excellence. Reflection, sharing, and modeling emerging, promising practices regarding effective use of 
technology for continuous growth, instructional gain and communication with stakeholders are examples of school 
leaders and administrators dedication to excellence in digital learning integration. Additionally, administrators will 
connect with and learn from teachers, administrators, and industry experts locally, nationally, and globally, evaluate 
emerging and current technologies for their potential to enhance learning environments. 

Overall, respondents did not perceive that they performed well in the area of personal growth and 
connectedness, responding that they needed to identify additional opportunities to improve their current personal 
growth and connectedness plan. Sixty-one percent of the respondents perceived that they only somewhat reflected, 
shared, or modeled effective use of technology for professional development. More than 50% of respondents stated 
that they somewhat connected with and learned from others, and somewhat evaluated technologies for potential uses. 

Community focus area. Administrators focusing on community will engage all stakeholders in the 
purpose and function of the school, leveraging multiple types and points of connection and communication to ensure 
the constant, effective flow of information and input. It is important to model responsible use of technology 
including, but not limited to, communication, social, ethical, legal, and global issues while facilitating and 
leveraging effective partnerships between the school and greater community, including local, state, and global 
communities, to improve the organization and opportunities available to staff and students in support of digital 
learning. School administrators will need to leverage online communication channels to create and maintain open 
discourse and collaboration with community stakeholders to establish and meet learning goals. Overall, the majority 
of respondents perceived that they did not do well in the area of community. Sixty percent of respondents perceived 
that they significantly or fully modeled responsible use of technology. Additionally, the majority of respondents also 
perceived that they only somewhat or minimally facilitate and leverage effective partnerships between school and 
greater community, as well as leveraging online communication channels. 
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Semistructured Interviews 


Six K-12 school administrators responding to the web-based questionnaire agreed to participate in a 
semistructured interview. Each of the open-ended questions posed supported the three research question. Examining 
school administrators perceptions about the NC Digital Learning Competencies for Administrators, lead to three 
common themes: (a) we do not know what the competencies represent; (b) we do not have the technology skill sets 
from educational training, and (c) we need models and examples of all levels of the competencies to be successful. 
Principal | stated, “Well first I think they too need to see examples of lessons because the principles don’t know. 
They really need a model to see model examples and I don’t know it would be too far-fetched to have them do 
artifacts”. Principal 3 supported what Principal | stated, 


How can you help them if you don’t know what’s out there, right? If they ask me a question about Power 
Teacher Pro, I don’t know because I don’t input grades. I can't help you. There has to be some knowledge 
base on my behalf on what they are using. To me, for me to truly evaluate any teacher - because how can 
you evaluate if you don’t know what they are doing with it. We need more of an evaluation how to find 
things. If I’m going to help with curriculum, which that’s what we’re preparing children for the 21st 
century, you just can’t jump on every app that comes along. 


Principal 4 agreed by purporting opportunities to assist with evaluation: 


Just learning, I mean learning the DLC that’s one thing I need to be more knowledgeable on. That just 
takes time to sit down and go through them because there is so much as administrators that we have to do. 
So having time and then you earmark here is time to do THIS - like we have to do things like Title 1 and 
school improvement plans - earmarking time to dedicate time to do learn them. 


Principal 3 went on to describe making the the competencies work, in spite of feeling a lack of confidence as a 
qualified leader in the Digital Learning Competencies. 


I make it work but I’m not the best qualified I could be because we don’t have to be - it wasn’t something 
we learned - it wasn’t in our curriculum. I think that new administrators coming out and the sign of the time 
they probably need it. They do need it but like I said, teachers help me a lot with that and I’m not 
embarrassed to say that, but they will say how do I do this but once they show me I’ve got it - you know - 
but that’s still not closing the gap if we don’t do something to prepare them. 


The second research questions asked NC school administrators’ perceptions about the NC Digital Learning 
Competencies for classroom teachers. Five themes emerged: (a) basic competencies for teachers, (b) district level 
supported tools (c) district level supported professional development, (d) a gap between current skills and necessary 
skills to meet teacher competencies, (e) technology use to curriculum integration. Administrators may not have the 
basic technology skills required by teachers to implement the digital learning competencies in the classroom. For 
example, Principal 2 stated, 


My lens is going to be a little different because my background was curriculum so I am not just looking: at 
does she know how to use a SMART board but do you know how to use an app. It’s more or less how are 
you applying that app and how are you able to reach kids in a different way through that technology. I 
really want to see what they're doing and it's not that I'm not just checking off a box that I saw you using 
that I want to see how it’s impacting instruction. 


Principal 1 described partnering with human resources to support teachers to not only understand the competencies 
but also how they earn continuing education units (CEUs) for professional learning opportunities. 
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I think that something at the district-level we have, you know, we have partnered with human resources 
and we are trying to work together on that that part of the really the rollout and in their expectations of 
having those CEUs. So I don't know that we're really where we need to be with that. Unfortunately, 
sometimes I get a phone call that ‘shouldn’t this count as a detail credit?’ So I don’t know, I have to go 
back in and look so. We probably need to have something a little bit more concrete that we can really say- 
‘yeah’ that counts; that doesn't count and we tried to do it but it's not where we need to be. 


Additionally, Principal 1 discussed demonstrating the leadership competency to support teachers. 


I think them having a real understanding of what it meant when you're looking at the leadership role we are 
looking at building capacity in your building I mean what does that mean in examples of what demonstrates 
that competency. I think that DPI has done a good job putting things out there. A lot of times, nothing 
against DPI, but a lot of times they just put things out there and just say do it. But with this particular 
initiative they did their homework for a change and they did provide a lot more resources, unlike other 
things in the past. Teachers think this too shall pass but I don’t think this is really going to. 


Principal 3 described the lack of consistency across competencies for the teachers: 


Well a lot of teachers think that just using it is meeting the competency. - using it is not competency. I use 
it to see how the students are engaging in the learning not just using it in general. PowerPoint using is not 
digital competency, it is how the students are using it to learn is digital competency. 


The third research question examined NC school administrators’ best practices assessing the NC Digital 
Learning Competencies for classroom teachers. Three common themes emerged: (a) current based practices are 
unknown, (b) administrators are not currently prepared to assess best practices, and (c) the current assessment 
practices utilizes a required rubric, which address Standard IV: C & D, which does not address NC DLCs for 
classroom teachers. Principal | stated, “.... so best practice in my eyes we should be consistent and we're not quite 
there yet.” Principal 2 explained the current evaluation instrument in more detail when responding to research 
question 3: 


Right now in the past it was just the teachers evaluation instrument - where evaluations or observations are 
done. We go to classroom to observe classroom teaching and learning there is a thing dealing with 
technology - Why did you choose to use those forms? - the walk through form is a district form - it is an 
app or a document. The teacher evaluation instrument is by the state of NC it is mandated. If there were 
other instruments or other ways to assess the teachers or evaluate teachers I would surely appreciate it - 
truly incorporate them. 


Based on responses from all 6 principals, there is not a standard assessment form for the DLCs for 
classroom teachers, and they would like to see some type of standardization. Additionally, professional development 
training would improve assessment practice and lead to improved staff development. “I could always use more 
training. Who couldn’t it? Cuz it’s always changing - you are never over trained in technology” (Principal 3). 
Principal 4 concurred with the need for additional training, 


Well - I think I can always learn because I am learning from them. The generation I am supervising help 
me because that’s not what we did when we were in college and in our staff development. We are getting 
information about the DLCs right ahead of the teachers. I do think that I would like to add a rubric or tool. 
It would be good for me to go back and take a few classes or to get or have a certificate or something to say 
that I did it; I graduated in my 30th year starting this month - and you think about that compared to 
someone that just got out of college and I’m evaluating them on technology; there’s a gap. They are digital 
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natives and that is an accommodation I as a leader in my area in my years that’s something - it’s probably 
not the best practice but it’s the best practice right now because I do try to stay ahead of the ball. 


Recurring themes emerged during the interview analysis process: (a) we do not know what the 
competencies represent; (b) we do not have the technology skill sets from educational training, and (c) we need 
models and examples of all levels of the competencies to be successful. The researchers could then see an emerging 
connection to the results from the web-based questionnaire. Respondents low perceptions of their personal growth 
and connectedness and their need to identify additional opportunities to improve their current personal growth and 
connectedness plan is further supported by their responses in their semi-structured interviews. Structured learning 
and digital training are supported by both interview and questionnaire analysis. 


Future Research 


The NC DLCT and NC DLCSA are relatively new. Currently, there are no standard practices for assessing 
whether the digital competencies have been met by classroom teachers. Additionally, assessment and evaluation 
practices differ significantly across the United States on whether classroom teachers are meeting ISTE technology 
standards. School administrators in the pilot study are aware of the standards and the need for additional training in 
order to meet state and national standards. Future research includes a web-based questionnaire with open and closed- 
ended questions to identify gaps between skills school administrators currently possess and the skills they will need 
to successfully implement the DLCSA and support the DLCT. Administrators, while identifying best practices for 
the annual evaluation process, will also connect to their current knowledge of evaluation and technology integration. 
Additionally, the administrators will have the opportunity following certification training to put this new information 
into practice at their school and/or district. Supporting administrators to evaluate classroom educators about the 
DLCs, will support classroom teacher practice, which could lead to increased student learning outcomes locally and 
statewide. Additionally, this topic is extremely relevant and can certainly be considered a real-world connection for 
the learners. A new certificate program in the state to support administrators’ views about how to evaluate Digital 
Learning Competencies for Educators or to meet the Digital Learning Competencies for Administrators could be 
beneficial. 
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Appendix A: The Digital Learning Competencies for Classroom Teachers 


se@ DIGITAL TEACHING & LEARNING 


= 
About the NC Digital Learning Competencies for Classroom Teachers 


The teaching and learning process is a complex balance of content knowledge, pedagogical strategies, and 
technological resources. The following Digital Competencies, informed by International Society for Technology 
in Education (ISTE), International Association for K-12 Online Learning (INACOL), and the NC Professional 
Teaching Standards, are to be viewed within the context of the current North Carolina Professional Teaching 
Standards as extensions in relationship with the ways that digital technologies impact and affect schools. 


Teachers and administrators should use these competencies to improve their practice and drive student 
learning within their classrooms. The following four Focus Areas have been loosely aligned to the Professional 
Teaching Standards with a subset of competencies that help to explain and ‘unpack’ the Focus Area. 


Leadership in Digital Learning 
Teachers will demonstrate leadership in accelerating their integration of digital teaching and learning 
pedagogies. 


Engage in virtual and face-to-face learning communities to expand mastery of technological applications for 
professional growth and student learning. 


Take initiative with own professional growth to inform practice. 
Demonstrate leadership for technology innovation beyond my own classroom. 


Engage in peer collaborative problem solving through continuous planning, designing, testing, evaluation, and 
recalibration of teaching methods using appropriate digital technology. 


Promote open, lifelong learning as an iterative process of success, failure, grit, and perseverance. 


Digital Citizenship 


Teachers will model and teach digital citizenship by the ethical, respectful, and safe use of digital tools and 
resources that support the creation of a positive digital school culture 


Demonstrate understanding of intellectual property rights by abiding by copyright law, intellectual property, and fair 
use guidelines. 


Teach and require the use of copyright law and fair use in student work and creation. 


Engage in responsible and professional digital social interaction. 
Integrate digital citizenship curriculum into student learning. 


Demonstrate global awareness through engaging with other cultures via advanced communication and collaboration 
tools. 


Ensure full, equitable access and participation of all learners through high-quality technology tools and resources. 
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Digital Content and Instruction 


Teachers will know and use appropriate digital tools and resources for instruction. 


Design technology-enriched learning experiences that encourage all students to pursue their individual interests, 
preferences, and differences. 


Lead all students in becoming active participants in setting educational goals, managing learning, and assessing their 
progress through digital tools. 


Identify, evaluate, and utilize appropriate digital tools and resources to challenge students to create, think critically, 
solve problems, establish reliability, communicate their ideas, collaborate effectively. 


Immerse students in exploring relevant issues and analyze authentic problems through digital tools and resources. 


Evaluate and appropriately modify the form and function of the physical learning environment to create a conducive 
digital learning environment. 


Data and Assessment 


Teachers will use technology to make data more accessible, adjust instruction to better meet the needs of 
a diverse learner population, and reflect upon their practice through the consistent, effective use of 
assessment 


Integrate digitally enhanced formative and summative assessments as a part of the teaching and learning process. 


Use performance data and digital tools to empower student metacognition for self-assessment & self-monitoring their 
own learning progress. 


Utilize multiple and varied forms of assessment including examples of student work products. 


Utilize technology and digital tools to synthesize and apply qualitative and quantitative data to: 
e Create individual learner profiles of strengths, weaknesses, interests, skills, gaps, preferences. 
e Inform, personalize, and calibrate individual learning experiences. 
e Identify specific plans of action related to weaknesses, gaps, and needed skills as identified in the learner 
profile. 
e Reflect and improve upon instructional practice. 
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Appendix B: The Digital Learning Competencies for School Administrators 


78 a: 5 DIGITAL TEACHING & LEARNING 


About the NC Digital Learning Competencies for School Administrators 


The following Digital Competencies are to be viewed within the context of the current North Carolina Standards 
for School Executives as extensions in relationship with the ways that digital technologies impact and affect 
schools. School and district administrators should use these competencies to improve their practice, build 
capacity in their staff, and drive student learning within their schools. Each Focus Area carries with it a subset 
of competencies that help to explain and ‘unpack’ the Focus Area. 


Throughout ail of the competencies is the underlying assumption of leadership and excellence with regard to 
digital citizenship. Administrators should model the behavior they expect from their staff and students and 
should continually seek to represent their schools and districts with the way they convey themselves both on 
and offline. 


Vision and Strategy 


Administrators will create and communicate a vision for digital teaching and learning in their schools, 
embedding into the strategic plan for implementation and execution 


Cultivate and articulate a clear and relevant vision and strategy for digital learning 


Advocate for, prioritize, and ensure equitable, sustainable access to available technology resources and encourage 
full participation of all learners in a digital learning environment (ISTE) 


Plan for and use funding effectively to support and sustain vision for digital learning 


Facilitate a school improvement planning process that is centered around personalized learning supported by digital 
learning environments 


Content and Instruction 


Administrators will be the ‘lead learners’ in their schools, modeling appropriate instructional practices and 
ensuring content encompasses appropriate digital tools, resources, and pedagogies 


Promote and model positive digital citizenship as well as practical policies for communication and collaboration with 
stakeholders to ensure responsible, effective digital teaching and learning practices throughout all school processes 


Actively advance and promote digital competencies for teachers by increasing access, opportunity, and resources for 
professional growth and the development or acquisition of instructional materials 


Establish and use systems to analyze and share data to guide whole-school and classroom-level continuous 
improvement 


Establish and use systems for the acquisition, vetting, creation, and implementation of digital content as well as 
evaluation systems for effectiveness 
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Human Capacity and Culture 


Administrators will leverage digital tools and resources to further develop a positive culture of learning that 
seeks continuous improvement among staff and students. 


Allocate time, resources, and access to support digital learning efforts, maximize capabilities of the school staff, and 
ensure ongoing professional growth for self and staff 


Provide learner-centered environments equipped with appropriate learning resources, including digital technologies, to 
meet the diverse needs of all learners 


Build technology, pedagogy, and content knowledge capacity in current staff members and create channels for the 
strategic recruitment of talented new hires 


Actively support staff through effective modeling and coaching practices, using relevant digital technologies to 
facilitate reflective two-way feedback 


Personal Growth and Connectedness 


Administrators will develop a personal leaming network and demonstrate a dedication for continued growth 
and excellence. 


Reflect on, share, and model emerging, promising practices regarding effective use of technology for continuous 
growth, instructional gain and communication with stakeholders 


Connect with and learn from educators, administrators, and industry experts locally, nationally, and globally 


Evaluate emerging and current technologies for their potential to enhance the learning environment 


Community 


Administrators will engage all stakeholders in the purpose and function of the school, leveraging multiple 
types and points of connection and communication to ensure the constant, effective flow of information and 
input. 


Model responsible use of technology including, but not limited to, communication, social, ethical, legal, and global 
issues 


Facilitate and leverage effective partnerships between the school and greater community, including local, state, and 
global communities, to improve the organization and opportunities available to staff and students in support of digital 
learning 


Leverage online communication channels to create and maintain open discourse and collaboration with community 
stakeholders to establish and meet learning goals 
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Appendix C: Web-based Questionnaire Results 


Participant Demographic 


Gender 80.95% Female n=17 
19.05% Male n=4 


Age 9.52% 21-30 years n=2 
23.81% 31-40 years n-5 
47.62% 41-50 years n=10 
19.05% 51-60 years n=4 
Job title 19.05% Principal n=4 
38.10% Assistant Principal n=8 
4.76% Digital Teaching & Learning Lead For District n=1 
19.05% Teacher n=4 
19.05% Instructional Coach n=4 


School location 76.20% Rural n=16 
23.81% Urban n=5 


Years of school administrator 57.14% Less than 1 year n= 12 
9.52% 1-5 years n=2 
14.29% 6-10 years n= 3 
9.52% 11-15 years n=2 
9.52% 15+ years n=2 


Web-based Questionnaire Questions; Vision and Strategy Focus Area 


Question Fully Significantly Minimally 


To what extent did you 5.88% 35.29% (n=6) 41.18% 5.88% (n=1) | 11.76%(n 
cultivate and articulate a clear | (n=1) (n=7) =2) 

and relevant vision and 

strategy for digital learning 


To what extent did you 41.18% (n=7) 11.76% 5.88% 
advocate for, prioritize, and (n=2) (n=1) 
ensure equitable, sustainable 

access to available technology 

resources and encourage 


effectively to support and 
sustain vision for digital 
learning 


To what extent did you 17.65% (n=3) 5.88% (n=1) | 11.76%(n 
facilitate a school =2) 
improvement planning 

process that is centered 

around personalized learning 

supported by digital learning 

environment? 


To what extent did you Plan 29.41% (n=5) 5.88% (n=1) ; 
for and use funding 
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Web-based Questionnaire Questions; Content and Instruction Focus Area 


all 


To what extent did you 16.67% | 33.33% (n=6) 38.89% 5.56% (n=1) | 5.56% 18 
promote and model positive (n=3) (n=7) (n=1) 

digital citizenship as well as 

practical policies for 

communication and 

collaboration with stakeholders 

to ensure responsible, effective 

digital teaching and learning 

practices throughout all school 

processes? 


To what extent did you 22.22% | 22.22% (n=4) 44.44% 5.56% (n=1) | 5.56% 18 
actively advance and promote | (n=4) (n=8) (n=1) 

digital competencies for 

teachers by increasing access, 


opportunity, and resources for 
professional growth and the 
development or acquisition of 
instructional materials? 


To what extent did you 16.67% | 38.89% (n=7) 33.33% 0.00% (n=0) | 11.11% | 18 
establish and use systems to (n=3) (n=6) (n=2) 

analyze and share data to guide 

whole-school and classroom- 

level continuous improvement 


To what extent did you 11.11% | 27.78% (n=5) 33.33% 16.67% 11.11% | 18 
establish and use systems for (n=2) (n=6) (n=3) (n=2) 

the acquisition, vetting, 

creation, and implementation 

of digital content as well as 

evaluation systems for 

effectiveness? 


Web-based Questionnaire Questions; Human Capacity and Culture Focus Area 


all 


To what extent did you allocate | 11.76%( | 41.18% (n=7) 29.41% 5.88% (n=1) 
time, resources, and access to n=2) (n=5) 

support digital learning efforts, 

maximize capabilities of the 

school staff, and ensure ongoing 

professional growth for self and 

staff? 


To what extent did you provide | 17.65%( | 35.29% (n=6) 41.18% 0.00% (n=0) | 5.88% 
learner-centered environments n=3) (n=7) (n=1) 
equipped with appropriate 

learning resources, including 
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digital technologies, to meet the 
diverse needs of all learners? 


To what extent did you build : 41.18% (n=7) 41.18% 
technology, pedagogy, and = (n=7) 
content knowledge capacity in 

current staff members and 

create channels for the strategic 


recruitment of talented new 
hires? 


To what extent did you actively : 35.29% (n=6) 35.29% 11.76% (n=2) | 5.88% | 17 
support staff through effective = (n=6) (n=1) 
modeling and coaching 

practices, using relevant digital 

technologies to facilitate 

reflective two-way feedback? 


Web-based Questionnaire Questions; Growth and Connectedness Focus Area 


all 


To what extent did you reflect | 5.56% 22.22% (n=4) 61.11% 5.56%(n=1) | 5.56% 
on, share, and model emerging, | (n=1) (n=11) (n=1) 
promising practices regarding 

effective use of technology for 

continuous growth, 

instructional gain and 

communication with 

stakeholders? 


To what extent did you 22.22%(_ | 16.67% (n=3) 27.78% 27.78% 5.56% 
Connect with and learn from n=4) (n=5) (n=5) (n=1) 
educators, administrators, and 

industry experts locally, 

nationally, and globally? 


To what extent did you 11.11%( | 11.11%(n=2) 55.56% 16.67% 5.56% 
evaluate emerging and current | n=2) (n=10) (n=3) (n=1) 
technologies for their potential 

to enhance the learning 

environment? 


Web-based Questionnaire Questions; Community Focus Area 


ly all 


To what extent did you model 11.11% | 50.00%(n= | 33.33% 0.00% (n=0) | 5.56%( 


responsible use of technology n=2) 9) (n=6) n=1) 
including, but not limited to, 

communication, social, ethical, 

legal, and global issues? 


62 


To what extent did you facilitate | 11.11%( 
and leverage effective n=2) 
partnerships between the school 

and greater community, 

including local, state, and global 
communities, to improve the 

organization and opportunities 

available to staff and students in 

support of digital learning? 


To what extent did you Leverage | 11.11%( 
online communication channels n=2) 

to create and maintain open 

discourse and collaboration with 

community stakeholders to 

establish and meet learning 

goals? 


0.00% 50.00% 
(n=0) (n=9) 


16.67%(n= | 44.44% (n=8) 
3) 
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27.18% 
(n=5) 


22.22% 
(n=4) 


5.56%( 
n=1) 
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Descriptors: Educational Robotics and Ethics 
Abstract 


Educators play a significant role in the formation of learner ethical perspectives and models; however, the 
ethical effects of the use of robots in education leave unanswered questions. Questions arise related to how ethical 
standards are determined and what model is used to emulate human ethics. Therefore, the purpose of this qualitative 
summative content analysis was to examine how the emulation of human ethics programmed in robots used in 
educational settings is perceived. A content analysis was used to explore this qualitative study. The main research 
question was How do robotics program developers perceive the role of emulation of human ethics when 
programming robots for use in educational settings? Educational leaders may use the findings in this study as a 
springboard for re-evaluating mission statements and ethical policies. Likewise, after reviewing the findings, 
robotic programmers may see the need for collaboration with educational leaders to engage in on-going discussions 
related to ethical standards, positive outcomes, issues, and challenges. 


Keywords: Robotics program developers, ethics, robots, education, programmed ethics, artificial intelligence 


Problem Statement 


Though scientific leaders strive to make advances robotics at a steady, ongoing pace, these leaders fail to 
fully investigate the wide-ranging implications of programming robots to emulate human ethics, especially in the 
educational setting (Ashrafian, 2015). Educators play a significant role in the formation of learner ethical 
perspectives and models; however, the ethical effects of the use of robots in education leave unanswered questions 
(Kubilinskiene, 2017). 

Questions arise related to how ethical standards will be determined and what model will be used to emulate 
human ethics. If educational leaders fail to evaluate, monitor, and play a dominant role in the creation of human 
ethics in robots, negative outcomes as well as some positive results may influence the determination of ethical 
standards for future generations (Hersh, 2014). When researchers’ findings and perceptions related to the use of 
human ethics to program robots used to promote learning are not examined, issues and challenges related to 
programmed ethics may tend to be underestimated or ignored (Bogue, 2014). Therefore, a content analysis of 
robotics program developers’ perceptions on the role of emulation of human ethics when programming robots for 
use in educational settings was explored in this qualitative study. 
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Purpose Statement 


The purpose of this qualitative summative content analysis was to examine how robotics program 
developers perceived the role of emulation of human ethics when programming robots for use in educational 
settings. A purposive sampling of online robotics program developer professional sites which focused on the role of 
emulation of human ethics used when programming robots for use in educational settings was included in the study. 
Content related to robotics program developers’ perceptions on educational uses of robots and ethics were analyzed. 

For study’s purpose, the qualitative summative content analysis was the most appropriate. In the qualitative 
method, perceptions on phenomenon are examined, and content is analyzed to find themes and patterns (Yen, 2003). 
A summative content analysis approach is appropriate when little is known about a topic and the identification of 
key words will be useful in the identification of themes or patterns (Nandy & Sarvela, 1997). In this study, a content 
analysis was used to examine keywords related to the phenomena of emulation of human ethics programmed in 
robots used in educational settings. No participants were interviewed. Thirty-two online robotics program developer 
professional sites which focused on the role of emulation of human ethics used when programming robots for use in 
educational settings were selected, and the content was analyzed. The phenomena were analyzed to find themes and 
patterns related to the perceptions of robotics program developers who shared online findings or implications on the 
role of emulating human ethics in robots in educational settings. 


The Researchers 


Four university doctoral faculty members conducted this content analysis study. After discussions on 
robotics and educational ethics, the researchers discovered a gap in the literature on the use of robots in the 
educational setting and the emulation of human ethics in robots. Therefore, to explore the implications for 
educators, they formed a group to research the topic to learn more about the topic. No personal gains resulted from 
the study. 


Literature Review 
Robotics in Education 


Robotics is a means of maximizing daily used skill sets of collaboration, problem solving, project 
management, and critical thinking to inspire individuals to get through mundane tasks at hand (Eguchi 2016). 
Robotics in education as a learning tool promotes knowledge at all levels, and the use of robotics helps engage 
learners in the learning process. In the literature, implications for the use of robotics in education includes findings 
to support the promotion of robotics to develop traits to be successful in a 21st century world of innovators, such as 
is demanded in a highly driven technology society. Educational robotics provides an environment conducive to a 
hands-on learning modality in which students confront critical thinking scenarios that inspire them to develop new 
and nontraditional solutions, tested/validated results using robotics, and a cyclical process used to come to 
successfully solution/resolutions problems of a more technical nature. Gura (2013) considered educational robotics 
to be “the most perfect instructional approach currently available” (para. 2). However, Gura noted that room for 
improvements in the learning environment must be made to accommodate transformation needed to meet 21* 
Century requirement. 

Accordingly, the successful adoption of an educational innovation such as the conception of robotics has 
enhanced the ability to make meet the true challenges. However, studies on developing curriculum content and how 
to incorporate the use of robotics in any educational framework is limited in the literature. Robotics is only one tool; 
therefore, the use of robotics must be aligned with correct theories of learning, educational philosophy, 
comprehensive curricula, and supportive learning documentation to ensure cohesive and successful approach 
(Shannon, 2015). Though limited studies exist on the role of emulation of human ethics programmed in robots 
identified educational challenges involved the need to make a shift from technology usage to collaboration of human 
intelligence in education while changing the pedagogy that embraces technology and learning theories, such as 
constructivism and constructionism (Kubilinskiene, 2017). 

In the literature, the consensus is robotic technologies should not be viewed as ordinary tools but as tools to 
promote the development of innovative ways to enhance the learning environment. Though few studies exist in 
which educational leaders’ perceptions on the role of ethics and the use of robotics are investigated, in the literature 
educators tend to agree that robotics learners engage in opportunities that enhance critical thinking skills (Julia & 
Antoli, 2016). Additionally, educators note the importance of students’ adaptation to a technical inspired society. In 
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existing findings, researchers demonstrated that students are afforded the opportunity to enhance technical abilities 
will augmenting learning strategies and problem solving within the 21‘ Century workforce requirements (Somyiirek, 
2015). 

In existing robotic studies, a paradigm shift from traditional education in which the curriculum guidance 
failed to adequately address the needs of technical requirements to the use of robotics throughout the curriculum is 
evident. Leaders show how the growth of robotic use in education changed the educational landscape (Rihtarsic, 
Avsec, & Kociancic, (2016). In the past, educators tended to view the use of technology in the classroom as 
mechanical instruments, which were referred to as black box technology. Black box technology is the use of 
computers, laptops, or [Pods to find information, engage in activities programmed to provide information, questions, 
and answers. With the advancement of robotics, black box technology tends to be less effective than white-box 
technology which is designed to stimulate higher levels of creativity and engagement than participating in black-box 
technology (Alimisis 2012). 


Artificial Intelligence 


Artificial Intelligence (AI) is a computer which functions based on human intelligence commonly referred 
toas AI. Artificial intelligence focuses on the integration of computer science, mathematics, psychology, cognition, 
and biology fields. In the literature, an extension number of studies have been conducted on how AI uses a range of 
knowledge from these fields to promote the rapid progress of emulation of human intelligence traits (Nimbekar 
2016). 


Educational Ethical Standards 


Few studies exist on educational leaders and ethical standards. In existing studies, study findings are 
limited and not directly related to educators’ perceptions on how ethical standards are viewed. Additionally, data on 
the impact of ethical standards on changes in education are scarce. Though ethical standards are included in policy 
sections of handbooks, educational leaders tend to focus on specific legalities rather than on innovative learning 
techniques and possible ethical issues related to learning outcomes such as the use of robotics or artificial 
intelligence. Two ethical standards are commonly outlined in most educational institutes. One area is instructor 
performance, and the second topic is centered on criminal-like behaviors (Zirkel, 2014). Generally, in educational 
institutes, ethical standards are defined and classified under three umbrellas: civil rights, specific local, state, and 
national laws, and contractual standards (Umpstead et al, 2013). 


Educational Ethical Frameworks 


With the infusion of technology in the 1990s, the educational learning environment changed rapidly, and 
educational leaders failed to keep up with the need to review and re-evaluate ethical issues and challenges related to 
technological advances and robotics (Bottino, 2016). Though changes may have included the emulation of human 
ethics in robots, in the literature, few studies are focused on ethical issues and the need for updated theoretical and 
conceptual frameworks for ethics in education. In the limited number of published studies, findings are indicators 
that a need exists for balanced ethical frameworks which address 21‘t Century technological issues and challenges in 
education (Martinov-Bennie, & Mladenovic, 2015). 

Educational leaders tend to select theoretical and conceptual frameworks which include moral decision- 
making theories and ethical guidelines that can be adapted for the educational setting (McBride & Hoffman, 2016). 
Kitchener (1984) created a decision making moral principles framework which addresses five basic principle-based 
guidelines. The guidelines include focus on individual autonomy, justice for all, caring for others, no maleficent 
intentions, and trustworthiness. Kitchener’s (1984) principles are highly favored by educational leaders. Though the 
framework was created before the emulation of human ethics was used by robots in educational settings, educators 
tend to create ethical policies which are based on Kitchener’s framework (Bates, 2004). 


Human Ethics in the Educational Setting 


Baxter et al. (2017) identified three areas for research when using robots in the classroom: Pedagogy: to 
enhance the teaching environment. Methodology: Try to prevent bias in the teaching environment caused by 
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inflating children’s expectations prior to the interaction with the robots. Ethics: What are the roles of robots in class? 
Is the robot a peer or someone (or something) that cannot be trusted? 

What should be the robot behavior in the classroom? Gelin (2013) cited Issacs Asimov three laws of the 
robotics suggesting that to be the norm “1- A robot may not injure a human being or, through inaction, allow a 
human being to come to harm, 2- A robot must obey the orders given to it by human beings, except where such 
orders would conflict with the First Law, 3- A robot must protect its own existence as long as such protection does 
not conflict with the First or Second Laws” (p. 69). These laws are easy to conceptualize, but programming robots is 
not easy, and those expectations can only be accomplished by programming robots that can act and behave like 
human beings (Jordon & McDaniel, 2014). 


Ethical considerations for the use of robots in the classroom 


The ethical use of robots to teach is an area that merits greater exploration and study. Since the 1950s and 
1960 when B.F. Skinner’s pioneered concepts related to teaching machines educators have contemplated the 
efficiencies that a computer or robot might provide to better optimize student learning in the classroom (Rutherford, 
2003). A constant question debated by roboticists is the extent to which robots should be designed to nudge a user 
in terms of their thinking and decision making (Borenstein & Arkin, 2016). For example, researchers have 
contemplated the effectiveness of humanoid robots within a socio-cognitive paradigm intended to support 
knowledge acquisition, like Skinner’s theory of teaching machines and programmed instruction, robots can provide 
forms of feedback and other learning advantages for students in the classroom (Mazzoni & Benvenuti, 2015). 

One recent comparative study of robots in school-based learning found that the use of an actual humanoid 
robot in the classroom provided qualitative advantages related to social impact over just using simulators (Bacivarov 
& Ilian, 2012). Robots potentially allow course and training designers to provide the addition of a humanoid 
embodiment along with tools for social interaction to the learning context beyond a PC based computer program. In 
addition, anthropomorphic features of humanoid robots along with characteristics related to repeatability, flexibility 
and digitization may be engaging as well as motivating to young students within the classroom (Chih-Wei et al., 
2010; Toh et al., 2016; Kazakoff et al., 2013). 

The ability of robots to make ethical decisions is also a consideration for their use in situations that involve 
humans. For example, when it comes to artificial agents (including robots) an emerging field of machine ethics has 
developed around a somewhat elusive goal of potentially programming ethical patterns of thought into systems that 
are being developed (Torrance, 2008). Moreover, roboticists continue to explore the prospect of robots 
autonomously making ethical decisions because machines do not have emotions and are unable to consider the 
emotions of individuals who may be adversely impacted by their actions (Anderson & Anderson, 2010). 


Method and Design Appropriateness 


Qualitative researchers explore phenomena by trying to make sense of them (Yin, 2003). In this study, online 
scholarly researchers’ perceptions of a phenomenon were analyzed. The quantitative method was not appropriate for 
this study. Quantitative researchers use numbers, instead of the examination of phenomena, to investigate the 
statistical significance of an event (Yin, 2003). 

Unlike in quantitative studies, numbers or statistics would not be significant when a researcher examines 
perceptions about an event such as the role of the emulation of human ethics programmed in robots used in 
educational settings. The use of numbers would be appropriate for a quantitative researcher who wanted to begin 
with a hypotheses and theories to describe a statistically significant comparison or correlation between types of 
robots and ethics. 

The design for this study was a qualitative summative content analysis. In summative content analysis, the 
researcher analyzes keywords related to a phenomenon (Nandy &Sarvela, 1997). In this study, the phenomenon was 
the emulation of human ethics programmed in robots. Articles selected to be analyzed in this study were published 
by robotics program developers who focused on robots and ethics in the education. In 2001, in the United States, the 
use of robotics in the educational setting increased. Science, Technology, Engineering and Mathematical (STEM) 
schools were established, and STEM leaders promoted the adaption and use of robotics in the STEM school 
curriculum (Holbrook, 2005). All articles in this study were posted online, and the public has complete access to the 
studies. 
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The case study or phenomenological design were not suitable for this study. Participants’ perceptions were 
not examined in this study, and interviews were not conducted. To identify themes and patterns, the focus was on 
the analysis of keywords related to robots and ethics in online articles selected by the researchers. Whereas, case 
study researchers examine participants’ perceptions on external events, and phenomenological researchers explore 
how participants feel internally about an event (Yin, 2003). In this study, a range of keywords related to robots and 
ethics were analyzed. 

The sampling method was purposive sampling, and 32 articles were selected for this study (Weber, 1985). 
To find themes and patterns, Weber (1985) recommended the use of at least 30 documents. Researchers conducted 
an inquiry on scholarly researchers’ perceptions on the emulation of human ethics programmed in robots used in 
educational settings. To obtain perceptions from the selected articles, the content analysis data method was used to 
collect data from 30 subjects: 10 in robotics, 12 in ethics and robotics, and 10 in ethics and robots used in 
educational settings. 

Informed Consent form were not appropriate for this qualitative summative content analysis. The 
researchers were the primary instrument in this qualitative study. Hsieh & Shannon (2005) defined qualitative 
content analysis as an organized approach to cipher and classify text content and analyze data. The content analysis 
approach was purposefully designed to aid the researcher in interpreting words, phrases, and sentences in a 
subjective manner. 

To code content analysis study data, five steps are appropriate (Weber, 1985). In this study, units to be 
used for recording such as words, phrases, sentences were established in the first step, and categories were created in 
the second step. The third step was the identification of samples to be used for data coding. During the fourth step, 
peer debriefing was used to test themes and pattern agreement. In the fifth step, content analysis was re-analyzed, 
and revisions were made. 

In this study, the researchers critically read the selected studies and took notes from each study. Keywords 
and phrases in each study related to the use of robots and ethics were carefully noted. For each study, a list was 
created to show different types of information discovered. List entries were descriptively categorized, and categories 
were closely analyzed to determine whether categories are related and, if so, how. Related categories were 
combined, and major themes, as well as minor themes, were identified. The major and minor themes were 
compared, and this process was conducted for each study. When major and minor themes were determined for each 
study, the major and minor themes found in each study were collected, compared, and contrasted. Similar major 
themes were merged, and similar minor themes will be combined. Careful analysis will be conducted to ensure 
relevancy and fit. After major and minor themes are identified, researchers will closely examine the content of the 
selected studies at least five times to assure a thorough content analysis. 

After the researchers analyze keywords and phrases for themes and patterns, NVIVO10 qualitative 
software was used to help the researchers identify and organize themes and patterns. In content analysis studies, 
triangulation is used for the collection of data from different sources, such as revision and retesting, use of raw data 
to compare interpretations of data, varied observations, and member checking (Lincoln and Guba, 1985). This study 
was triangulated though rich thematic descriptions. Quotations from articles were used to support themes. For 
member checking, researchers re-checked words, phrases, sentences, categories, themes, and patterns for the content 
analysis of each study included in the study, which means that data in each study were analyzed by four researchers. 
After each analysis was completed, researchers conducted a peer debriefing, which acted as a measure to test 
agreement on themes and patterns. The study was not generalizable to other content analysis studies. In a qualitative 
study, the sample is small and not as generalizationable as in quantitative studies. 


Research Question 


The central research question was used to guide the study. The qualitative researcher’s findings at the end 
of the data collection and data analysis provide answers to the central research questions and the sub-questions 
(Adler & Adler, 1987). In this study, the central research question was How do robotics program developers 
perceive the role of emulation of human ethics when programming robots for use in educational settings? Open- 
ended sub-questions will be the following. How do robotics program developers perceive the role of human ethics? 
How do robotics program developers perceive the role of human ethics in educational settings? How do robotics 
program developers perceive the role emulation of human ethics in robots? 
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Conceptual Framework 


Educational leaders are concerned with moral decision making and how to establish ethical guidelines for 
all educational stakeholders (McBride & Hoffman, 2016). Kitchener (1984) created a decision making moral 
principles framework which addressed five areas. The first principle deals with the rights of autonomy and how 
individuals act. In the study, findings on robotics in education and the emulation of human ethics will be explored. 
Themes and patterns that result from content analyses may include themes related to robotic programmers’ ethical 
considerations and how the educational community perceive ethical coding, as well as how students view the role of 
ethical emulation. Kitchener argued individuals have the right to act in the way they deem appropriate, unless they 
impede the well-being of others. 

In the second principle, individuals do not have the right to cause “physical or psychological harm” to other 
people (Kitchener, 1984). Articles explored in this study focused on the role of the reproduction of human ethics in 
robots used for educational purposes and psychological implications for students. Robotics program developers’ 
views on robots, ethics, and dangers of physical harm to students in the classroom were examined. 

The third principle is focused on helping other people. Individuals are responsible for the development and 
“enhancement of welfare of others” (Kitchener, 1984). In this study, the examination of robotics program 
developers’ perceptions provided insights on how ethics were programmed in robots. Keywords were analyzed to 
find themes related to specific categories and outcomes, which included categories on robotic behaviors and data 
which influenced ethical decision making. 

The Golden Rule is the basis for the fourth principle. All people must be treated fairly and equally 
(Kitchener, 1984). The role of human ethical imitations in robots used in education may be a theme that results in 
this study. Equality is an ethical goal included in educational mission statements, and the role of fairness and the 
use of robotics in the classroom may be examined by researchers. If equality for all students is not a resulting 
theme, educational leaders may use the study as a guideline to re-evaluate mission statements and the use of robots 
in the classroom. 

Trust is the foundation for the last principle. All persons must respect others, be truthful, and seek to keep 
their words (Kitchener, 1984). When robotics is used in the educational classroom, educational leaders must trust 
robotic program developers who create ethical standards for robotic-based learning. How trust is perceived was a 
pattern which arose in the content analyses of this study. 

The principles are idealist; however, in real world ethical situations, issues and dilemmas are constantly in 
flux. No constant standards can be maintained. Thus, applications may vary, and principles may be forsaken if a 
“higher moral purpose” is sought (Kitchner, 1984). Based on educators’ lack of knowledge on ethical designs for 
programming robots, leaders may need to analyze each moral principle area to evaluate which principles may need 
enhancement, clarification, or a new direction when considering ethical standards for the emulation of human ethics 
programmed in robots used in educational settings. 


Findings 
Theme 1: Computational Educational Moral Agents Verses Human Educational Moral Agents 


Theme | was computational educational moral agents verse human educational moral agents. In this study, 
robotics program developers agreed that educational robots were moral agents who made moral judgments based on 
a multitude of collected right and wrong ideas. Robotics program developers defined the educational robotic moral 
agent model as “a moral decision-making procedure modeled by a computational process.” During the moral 
decision process, educational robots are designed to search a wide range of online research-based frameworks and 
accountability levels, and highly effective moral decisions are made by matching individual situations to research- 
based data. Other programmers defined educational robots “...as better moral creatures which can make decisions 
more consistently that humans.” A third of the robotics program developers contended robots used in the 
educational setting act as reliable data based moral agents who lack inclinations to make ethics decisions based on 
cultural bias. One developer explained the differences between robots and human educators by noting that “age, 
ethnicity, educational levels, gender, and personal preferences play major roles in how humans view themselves as 
moral agents.” Computational educational moral agents verse human educational moral agent theme was a major 
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argument which influenced robotics program developers’ perceptions on the use of robots in educational setting and 
the emulation of human ethics. 

The robotics moral agent model decision making process espouses rational, unbiased decision making. In 
the computational educational moral agents’ model, robotics program developers found that educational robots 
minimized possibilities of personal or cultural influences. According to one programmer, “Unlike human based 
ethics, robot computational moral reasoning decisions generate from globally recognized and accepted 
philosophical, educational, and psychological critical thinking and problem-solving frameworks.” In the 
computational moral education agent model, educational robotics program developers tend to eliminate human 
based ethical decision making based on personal choices and environmental influences. 

Robots as bias free moral agents will affect how future generations view ethics. One programmer posited 
robotics educational program developers “are actually changing society,” and the robotics moral agent change model 
will determine how ethics are taught in educational settings. To help developers understand the impact of the 
computational moral agent model, other programmers suggested that educators should work in conjunction with 
computer science and robotics education program developers to teach those educators a basic understanding of 
human ethics. Unbiased robots acting as moral agents will act as generational change agents. 

Though the question of robotic moral agency and status raises questions for educators, benefits of using 
robots in educational settings outweigh the concerns. One group of programmers outlined the potential benefits of 
robots acting as moral agents to support both individual and collaborative learning activities exceeds concerns 
related to traditional, often non-situational, educational more agent frameworks. Robotics program developers 
acknowledged educators’ questions related to the ability of robots to make teacher decisions and control children’s 
behaviors in the classroom. However, given the vast array of case studies and research from which robots made 
decisions, robotics program developers perceived robot teachers as being highly capable of moral decision making. 


Theme 2. Perceived Lack of Robotics Accountability 


Theme 2 focused on a perceived lack of robotics accountability needed to maintain ethical standards and 
privacy in the educational setting. Because computational searches involve data-based searches of an infinite amount 
of material, robotics educational program developers noted they do not attempt to design, regulate, or monitor 
specific robotic accountability levels. After reflecting on how robotics accountability levels, one programmer 
pondered that an ongoing question which robotics program developers must consider is in terms of accountability 
“Can the robot be used as a form of human replacement in the classroom?” When discussing accountability and 
privacy classroom concerns related to vulnerable populations such as the poor or disabled, other programmers asked 
a similar question. Summing up robotics program developers concerns on the perceived lack of robotic 
accountability and privacy, a programmer noted that information stored on robots might be “subsequently accessed 
by others.” Robotics program developers recognize developers must continue to ask questions and monitor how 
accountability and privacy levels are perceived. 

In robotics, accountability is based on predetermined rigid scales. Educational robotics program developers 
indicated computational ethics robots are programed to utilize flexible, situational moral agency skills. Several 
programmers posited robotic brains rely on basic problem solving and critical thinking accountability models, and 
educational robotic ethical accountability depends on outcomes supported by computational theoretical frameworks, 
case studies, and research-based studies searches. Another programmer noted human are “accountable for actions 
taken and having the ability to individually make right and wrong decisions.” When educators perceive robots in 
human terms, accountability levels appear diminished. 

Lingering, unanswered accountability questions and educator perceptions on the importance of establishing 
accountability levels create concerns about the use of robots in the educational setting. Robotics developers 
identified specific questions of concern, and they acknowledged why educators perceived a possible lack of 
accountability. One programmer agreed that in the classroom “ethical regulations and traceability of robot actions is 
of concern.” Because robotics program developers program robots to serve as classroom tutors, tools, or peers, F7 
believed the answer to the question that asks, “Who will bear ethical and social responsibility for robot behaviors” 
has not been determined. Ongoing, unanswered accountability questions increase concerns related to the use of 
robots in classrooms. 

Designs of robots used in the classroom impact accountability perceptions. Robotic program developers 
perceive a positive relationship between classroom robots and students. F29 explained, “Computationally, 
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educational robots are designed to appear as if they understand and care for learners and masquerade as friends and 
companions” by promoting appropriate social and ethical behaviors. Because robots are programmed to emulate 
caring, ethical behaviors, students tend to perceive a genuine affection. F12 emphasized that “children tend to form 
strong attachments to robot companions and robot teachers.” Learners’ positive perceptions and attachments to 
robots create a high level of accountability expectations. 


Theme 3: Robots as Non-Human 


The third theme was robots as non-human. Traditionally, educational ethics are defined by a set of rules 
established for all stakeholders in all situations. Robotics program developers visualized the role of ethics through a 
different lens. Programmers emphasized that “robots are not humans,” and robotics objectives are not based on the 
emulation of human educational ethics. F16 firmly acknowledged that robots emulate ethics based on a “multitude 
of research, theoretical frameworks, and case study situations.” Robotics program developers believed the use of the 
computational process stimulated increased flexibility in ethical decision making. F9 noted a need for ethical 
decision making which is “flexible in a changeable environment.” Robotic program developers perceived a need for 
a more flexible ethical decision-making process. 

Emulation of human ethics to program non-human robots used in the educational setting is not a robotics 
program developers’ objective. To help educators understand the focus on non-human ethical objectives, robotics 
programmers created “roboethics” roadmaps for educators. According to F25, the roadmaps will “provide a 
systematic assessment of the ethical issues involved in Robotics R&D, increase understanding of the problems at 
stake, and promote further studying and transdisciplinary research.” Robotics program developers recognize 
that robots are non-human, and the emulation of human ethics does not meet robotic objectives. 

Non-human moral agents play a distinct and needed role in the educational setting. Robotic developers 
argued the importance non-human robotics moral agency. F11 posited, “Robot ethics depends upon the notion that 
robots might in some sense be moral agents in their own rights.” Educational robots are accountable for 
recognizing, synthesizing, and processing well-known and highly appraised models which are the most appropriate 
for ethical situations. The role of non-human moral agents in educational settings is a role which robots can fulfill. 


Findings Summary 


The study findings answered the main research question. The main research question was How do robotics 
program developers perceive the role of emulation of human ethics when programming robots for use in educational 
settings? Robotic program developers perceived robots in the educational setting as non-human moral agents. 
Therefore, the role of robotic program developers was not to emulate human ethics in the educational setting. 
Robotics program developers viewed the need for flexible ethical decision making in the educational setting, and, to 
avoid the emulation of human ethics, they perceived the role of robots was to use a multitude of data base searches 
of research, theoretical frameworks, and case studies for ethical decision making. 

The findings addressed the first sub-question How do robotics program developers perceive the role of 
human ethics? Robotics program developers viewed the importance of situational human ethics interpretations and 
implementations. To facilitate flexibility, robotics program developers programmed robots to search computer-based 
ethics related research, frameworks, and case studies. 

For the second sub-question, How do robotics program developers perceive the role of human ethics in 
educational settings?, robotics program developers acknowledged the importance of human ethics, but they felt 
more flexibility was needed in the role of how classroom human ethical models were created, developed, and used. 
However, some robotic program developers expressed questions and concerns about the implementations of flexible 
robot ethical accountability levels and behaviors in the educational setting. 

In response to the last sub-question on how robotics program developers perceive the role emulation of 
human ethics in robots, robotics program developers did not perceive that a role existed. Robotics program 
developers argued that educational robots were not designed or programmed to emulate human ethics. 


Limitations, Significance, and Implications of Study 


One limitation of the study was 32 online, public articles written by robotics program designers were 
analyzed through qualitative content analysis to find themes and patterns. In qualitative content analysis studies, 
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findings may not be as generalizable as in quantitative studies. Another limitation was only a limited number of 
articles written by robotics programs existed which addressed robotics and emulation of human ethics in the 
educational setting. 

The significance of this study is the need for a renewed global initiative in education to promote debates, 
research, and on-going collaboration with scientific leaders on ethics and programming robots. The implication for 
education leaders is to provide ongoing professional development on the role of ethics in education and to create 
best practices for using robots in education to promote increased student learning and enhance the teaching process 
(Vollmer, 2016). 

Traditional and widely accepted educational conceptual frameworks are idealistic, and ethical models 
related to the use of robotics are non-existent (Hersh, 2014). Educational leaders must initiate a change program 
which will include the re-evaluation of mission statements and the possible implications of the role of emulation of 
human ethics by robots used in the education setting. During the re-evaluation, issues of possible “physical or 
psychological harm” to students and the learning community must be examined, as well as how the use of robots by 
educators may benefit learners (Kitchener, 1984). 

Change programs must include all stakeholders who understand the need for investigating and establishing 
high levels of expectations for the emulation of human ethics by robots. Leaders must convince stakeholders that a 
moral reason for change is needed, and leaders must demonstrate how change will benefit global educational 
leaders, instructors, students, parents, the community, and robotic programmers. A trusting foundation for 
collaboration among all stakeholders, including robotic programmers, must be built, and methods for on-going 
communications must be established and developed (Fullan, 2006). 

The implications of this study are global. All cultures will be affected by the robotics’ shift in how students 
are taught ethical decision making in the educational setting. Robotics program developers will create computational 
educational moral models which will replace archetypal educational ethics frameworks. Because robotics program 
developers do not classify robots as human, educators, parents, and communities will continue to question the use of 
robots in educational settings, and they will challenge robotics ethical dilemmas, moral standards, and computational 
findings. The examination of robotics program developers’ perspectives through different lens may help close the 
gap and establish a new understanding among all stakeholders (Borenstein & Arkin, 2016). 
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Abstract 


When assigned to develop higher education courses for the online learning format, faculty members and 
instructional designers are often assigned to work together as a course development team. Sometimes, faculty 
members may be unaware of the field of instructional design and the valuable knowledge instructional designers can 
bring to a course development project. As a result, they may not realize that the advice and assistance instructional 
designers offer can help bring their courses to the next level. Instructional designers possess specific knowledge of 
learning theories and instructional design models that are the keys to improving the quality of instruction within 
online higher education courses. When such specialized knowledge is not utilized, the result can be low-level 
courses in which students are unsuccessful. Therefore, it is important for individuals within academia to begin to 
understand the key role instructional designers play in improving the quality of online higher education courses. 
Instructional designers often are responsible for helping faculty members write course objectives, create engaging 
assignments for the online format, and develop methods for presenting course information to learners. This 
presentation outlines research and information gathered from 12 research study participants that details the 
important role instructional designers play in course production and seeks to bring new knowledge about 
instructional design to the forefront of the field. Instructional designers are a valuable resource within higher 
education, and the expectation is that others within the field of academia will gain a clearer understanding about the 
need for instructional designers to be involved. Such an understanding can lead to a smoother course development 
process and a higher quality online course result. In addition to discussing the role of instructional designers in 
higher education, the 12 research study participants shared their strategies for successfully working with faculty 
members to develop high-quality courses within higher education. 


Importance of Instructional Designers in Online Higher Education 


When assigned to develop higher education courses for the online learning format, faculty members and 
instructional designers are often assigned to work together as a course development team. Sometimes, faculty 
members may be unaware of the field of instructional design and the valuable knowledge instructional designers can 
bring to a course development project. As a result, they may not realize that the advice and assistance instructional 
designers offer can help bring the faculty members’ courses to the next level. Instructional designers possess specific 
knowledge of learning theories and instructional design models that are the keys to improving the quality of 
instruction within online higher education courses. When such specialized knowledge is not utilized, the result can 
be low-level courses in which students are unsuccessful. Therefore, it is important for individuals within academia 
to begin to understand the key role instructional designers play in improving the quality of online higher education 
courses. 

This paper outlines research and information gathered from 12 research study participants that details the 
important role instructional designers play in course production and seeks to bring new knowledge about 
instructional design to the forefront of the field. Instructional designers are a valuable resource within higher 
education, and the expectation is that others within the field of academia will gain a clearer understanding about the 
need for instructional designers to be involved. Such an understanding can lead to a smoother course development 
process and a higher quality online course result. In addition to discussing the role of instructional designers in 
higher education, the 12 research study participants shared their strategies for successfully working with faculty 
members to develop high-quality courses within higher education. 
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Importance of Quality in Higher Education 


Mykota (2013) stated that over the past few years there has been a large increase in the number of North 
American students enrolled in fully online education courses. The leadership teams within most higher education 
institutions now believe that their futures depend on their ability to provide quality online learning environments 
(Mykota, 2013). Allen and Seaman (2011, 2016), however, suggested that a large amount of lower quality courses 
being produced for the online format has begun to undermine the value of the educational opportunities afforded by 
the Internet. Therefore, it is important to ensure that each course produced is of the highest quality possible. 

What exactly is considered a high-quality online course? A high-quality online course can be defined as a 
course in which students perform well, are motivated to learn, and remain engaged in their coursework (Cole, 
Shelley, & Swartz, 2014). Students must feel a sense of satisfaction and must believe that they have truly learned 
something valuable when they complete a course for the course to be considered high-quality (Cole, Shelley, & 
Swartz, 2014). Research has shown that a successful online course development project involves not only a faculty 
member but also an instructional designer who has knowledge of the pedagogy involved in designing a course for 
the online format (Brown, Eaton, Jacobsen, Roy, & Friesen, 2013; Outlaw, Rice, & Wright, 2017; Stevens, 2013). 


Instructional Designers in Higher Education 


While instructional designers may not know much about fire science or criminal justice, they are specially 
trained to take the expert content given to them by faculty members and transform it into learning experiences that 
will capture students’ attention so that they can achieve the knowledge and skills necessary to be successful in their 
chosen careers. Instructional designers perform many different duties when working with faculty members to design 
and develop courses. They take on many of the behind-the-scenes responsibilities within the course production that 
sometimes faculty members are unaware of so that they can focus on providing the expert content that students need 
to be successful. For example, instructional designers have been known to transcribe videos and audio files, develop 
various forms of media for courses, and even load courses into learning management systems. 

However, there is much more to being an instructional designer than those types of activities suggest. 
Instructional designers today possess specific knowledge of learning theories and instructional design models that 
are the keys to improving the quality of online higher education courses (Shaw, 2012). They often are responsible 
for helping faculty SMEs write course objectives, create engaging assignments for the online format, and develop 
methods for presenting course information to learners (Hixon, 2008). Instructional designers are also often 
instrumental in helping higher education faculty negotiate and reduce the transactional distance that often occurs in 
online courses (Lunce & Huang, 2013). Transactional distance involves the misunderstanding and 
miscommunication that can occur between a learner and his or her professor due to the two parties being physically 
separated from one another (Lunce & Huang, 2013). 

Brigance (2011) stated that higher education institutions offering online learning need individuals with a 
clear understanding of the direction and approach that needs to be taken to produce high-quality online courses and 
that instructional designers possess just that type of understanding. Instructional designers possess the following 
attributes that are necessary for bringing online courses to the highest levels: 

solid designer foundation in instructional and learning theories, 

an understanding of the cognitive process of learning, 

ability to utilize research to inform practice, 

competency in multimedia and online educational formats, and 

commitment to perpetual learning and readiness for challenges along the way (Brigance, 2011; Fyle, 

Moseley, & Hayes, 2012) 


Instructional designers understand the need to keep up with a constantly changing field and the importance 
of working collaboratively with faculty members (Anderson, 2012; Brigance, 2011; Fyle et al., 2012). Some of the 
skills instructional designers possess that lend themselves well to improving the quality of online courses they help 
to produce include the following: 

possession of effective communication skills, 

knowledge of the need to constantly update instructional design skills, 

ability to apply current research and theory, and 

ability to identify and resolve ethical and legal issues (Anderson, 2012; Brigance, 2011; Fyle et al., 2012) 
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In a recently completed research study, 12 instructional designers who had worked within the field of 
online higher education for at least two years and had worked with faculty members to produce courses at least five 
times were interviewed. These instructional designers were asked if they believed that instructional designers were 
important within the field of higher education and, if so, how. Each of the participants agreed that instructional 
designers are important because they truly make a difference in the quality of the courses being produced for the 
online format. Instructional designers bring a specialized knowledge of instructional theories and how people learn 
to the table of online course design. Participants additionally stated that instructional designers help faculty members 
present course material in more engaging ways that can help students better absorb the subject matter. Instructional 
designers can also help faculty members keep the level of the students’ knowledge about the subject matter in mind 
and can bring fresh eyes and new perspectives to course design that can help to improve how students experience 
online courses within higher education institutions. Several participants mentioned that faculty members often do 
not realize that the way a course is presented in a traditional classroom must change when it is transferred to an 
online format, and instructional designers bring a wealth of knowledge regarding such a change in pedagogy. 

All but one of the participants touched on the fact that while faculty members are experts in their own fields 
of study, many do not have the background in education that instructional designers do. As a result, it can sometimes 
be difficult for faculty members to present their subject matter knowledge in ways that students can truly 
comprehend, especially in the online format, which can reduce the quality of the resulting course. One issue that a 
participant raised was that traditional faculty in higher education, while being trained specifically about their 
content, are often not trained as educators. Instructional designers can help faculty members translate their 
knowledge into a learning environment in which students can learn and achieve the outcomes of the course. In 
addition, the participant indicated that instructional designers are necessary to show the faculty different aspects of 
teaching and to help them deliver their content in ways that students can truly learn and understand, which is a 
hallmark of a high-quality online course. 


Other participants described in more detail the specialized knowledge that instructional designers bring to an 
online course development project, which include the following: 

Instructional designers are trained to specifically think about the student experience as they are working to 
develop online courses. They ensure that the subject matter is presented at the proper level so that students 
can absorb the material, especially if it is the first time they have ever encountered a topic. 
Instructional designers can make sure that instructions for assignments are clearly written so that students 
understand what is being asked of them. 
Instructional designers ensure that all the course material aligns with the course outcomes or objectives. 
Instructional designers retain consistency in how courses are presented in their institutions because they 
know that students are more successful in a course in which they know where locate specific files or 
content that they need to complete their assignments. 
Instructional designers bring a new perspective to the process of course design. They do not have an 
emotional or personal investment in a specific course development method and can look at course material 
with a different, more detailed, eye. Instructional designers are better able to see the issues that are keeping 
students from being successful in the course and can offer suggestions to faculty members that can help 
them improve their courses. 


Strategies for Helping Others Realize the Importance of Instructional Designers 


The outcomes of this study indicate that the development of a high-quality course for the online higher 
education format requires the expertise of more than one individual as well requiring the use of different types of 
teaching and learning strategies (Chao, Saj, & Hamilton, 2010; Vandenhouten, Gallager-Lepek, Reilly, & Berg, 
2014). Faculty members bring an extensive knowledge of the subject matter covered in a course, and instructional 
designers bring specialized knowledge of how to present the subject matter in such a way that it helps students 
achieve the outcomes and goals of the course. To produce a high-quality course, these two individuals must be able 
to collaborate well with one another and form a cohesive team based on mutual respect for one another’s time and 
expertise. 

The instructional designers interviewed for this study indicated that many of the issues that they have with 
faculty members stem from the fact that faculty members are often not aware of what instructional designers do and 
what benefits they can bring to online course design and to the field of higher education in general. The belief of 
study participants and researchers alike is that more information needs to be disseminated regarding the significant 
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contributions and advancements that have been brought into the field of higher education through the work of 
instructional designers (Afsaneh, 2014). 

The findings of this study also indicate that instructional designers and faculty members should make more 
of an effort to communicate with one another as communication appears to be the key to establishing a true course 
development partnership (Anderson, 2012; Ashbaugh, 2013; Campbell, Schwier, & Kenny, 2009). A successful 
partnership between a faculty member and an instructional designer can improve the quality of the resulting course 
and demonstrates how important an instructional designer can be within the field of higher education. Some 
strategies for improving the working relationship between instructional designers and faculty members include the 
following: 

Establish an initial meeting between the two before the actual course design work begins (Tessmer, 1993). 

This gives both parties the opportunity to get to know one another, learn about each other’s working styles 

and preferences, and hammer out issues regarding deadlines and methods for meeting the goals and 

outcomes of the course to hopefully avoid conflicts down the road. 

Maintain regular communication between the instructional designer and the faculty member. Several 

participants mentioned that conflicts arise when faculty members and instructional designers lose touch 

with one another during a course development project or when deadlines are not met. 

When offering feedback to faculty members, it is often better for an instructional designer to do so through 

a phone call or face-to-face so that the two parties have a chance to discuss the feedback together. This 

allows the instructional designer to explain more clearly why he or she thinks a change should be made, 

and it allows the faculty member the chance to give his or her opinion about the change. There are then no 
misunderstandings about the intent of the feedback, and each person has a chance to weigh in on the issue 
at hand, reducing the chance of conflict down the road. 

Instructional designers should present themselves as helpers or as individuals who can complement the 

abilities of faculty members. By doing so, instructional designers are more likely to be seen as equal 

partners in course production, which can help to raise their credibility in the eyes of higher education 
faculty. 


Instructional designers should make every effort to build a culture of teamwork with faculty members 
because it is teamwork and collaboration that will make the courses being produced the best they can be. Some ideas 
for creating a culture of teamwork include the following: 

Instructional designers should clearly communicate their role and purpose in the course development 

process. 

Instructional designers should put together a list of services that they can perform for faculty members and 

maybe even have a portfolio online where they can show faculty members examples of some of the work 

they have done and how this work improved the quality of the courses on which the instructional designers 
worked. 

Instructional designers should listen to the ideas that faculty members have for their courses and advise 

them on how an instructional designer can help them achieve those ideas. 

Instructional designers should have a thick skin and be prepared for faculty members not to accept the latest 

and greatest instructional technique that an instructional designer is suggesting. Instructional designers 

should be prepared to compromise with faculty members when necessary. 

Instructional designers should have knowledge of their university’s instructional design processes and have 

documentation in place so that they are always ready to answer any questions faculty members may have 

during course production. 

Instructional designers should let faculty members know that they are there to guide the faculty members 

through the course development project, not dictate to them. 

Above all, instructional designers and faculty members should together consider the needs of their students 

first and foremost and work together to ensure that all of the courses they develop are of the highest quality 

and provide a unique and fulfilling learning experience for all students. 


Conclusion 
High-quality online higher education courses are clearly the result of true collaboration and teamwork 


between faculty members and instructional designers, and more of an effort should be made by higher education 
leadership to promote this partnership in the future (Kotter, 2008; Kowch, 2009). Participants and researchers 
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believe that if more of an effort is made to clearly establish the roles and responsibilities of faculty members and 
instructional designers to course development projects, conflicts will diminish, and better courses will result which, 
in turn, will lead to a better outcome for students and more growth in the field of online higher education. 
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Abstract 


Creativity is often thought of as only belonging to the most privileged. This study is an examination of 
creative capability of students of different income levels. Eighth grade students from six middle schools in a 
Minnesota school district were tested using the Visual Format of the Torrance Test of Creative Thinking (TTCT). 
Data was also gathered regarding individual participation in the Federal Free or Reduced Price meals program (FRP) 
as an indicator of family income. Significant differences in performance were found between students classified as 
FRP eligible and non-eligible in four of six test areas. This observation will allow schools to better address the need 
for opportunities to assist low income students in becoming more creative. 


Introduction 


Creativity is often thought of as only belonging to those with schooling, opportunity, or with the 
availability of the arts. 995 8th grade students 13 to 14 years old from six middle schools were tested using the 
Visual Format of the Torrance Tests of Creative Thinking. Data was collected and anonymized by the district 
research department, and additional demographic and academic information was included in materials provided to 
researchers. This study involved dividing the data by participation or non-participation in the Free and Reduced 
Price meal program, an indicator of low family income. 

Why does creativity matter? “Creativity is a fundamental component of the human experience and is 
generally regarded in many societies and cultures as being among the highest qualities of human performance and 
character, as expressed through art and science” (Collard and Looney, 2014). Plucker (2010) has found creativity to 
three times stronger and indicator of lifetime success as is intelligence. Unfortunately, not everyone has the 
opportunity to become a creative person. People who are born into situations that undermine their way of life 
affecting their creativity are in danger of not developing creative skills. 

The study of creative development seeks insight into ways for individuals to become better problem-solvers 
and promote their ingenuity. “Creative thinkers tend to be innovative, explorative, can educate others, and are 
generally highly targeted by businesses. Highly creative people are good at problem solving, and problem-solving 
capability has been used to measure creativity of individuals in the past” (Hirschman, 1980). Creativity does not 
only benefit the individual in the long run, in adult life, but creative development on a higher scale will promote the 
wellbeing of our population. 

The purpose of this study is to investigate how the effects of socioeconomic status are evident in measured 
creativity. The Torrance Test of Creativity (TTCT) was be the instrument used to measure the creativity of students. 
The Free or Reduced Price program (FRP) represents students who are from low income households. The tested 
participants were divided into two groups, consisting of students who were eligible and those who were not eligible 
for the FRP program. A mean score from each of the six components between our two variable groups was 
determined and a T-Test was conducted to determine the P-value of each. The difference in means indicates the 
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relationship of performance in terms of creativity between groups. T-tests were used at .05 to look for significant 
differences between groups on each component. 

Although research has examined the effects of ethnicity, gender, and intelligence on creativity, little 
published research has examined the relationship between socioeconomic status and creativity. In this study, the 
goal was to find out if creativity scores are affected by socioeconomic status. This is a question of people's lives, 
beyond the value of their homes: The results of creativity enrich the culture and so they indirectly improve the 
quality of all our lives” (Csikszentmihalyi, 1997). 


Literature Review 


Research into creative development began with J.P. Guilford's charge to the American Psychological 
Association in 1950, which was significantly advanced by Paul Torrance. “Ellis Paul Torrance, the creator of the 
Torrance Test of Creative Thinking, has stressed the importance of developing children’s creative abilities since 
society advances most effectively through the contributions of its creative individuals” (Kim, 2006). The 
development of creative skills remain an important element of education. “Testing for creativity is considered a vital 
part of the educational assessment, mainly because it is valued in modern technological society and should be 
developed in as many individuals as possible” (Bart & Hokanson, 2014). 

The Torrance Test (TTCT) is a widely used tool to determine an individual’s creativity and is considered 
the best form of measurement for creativity to date. “The TTCT is the dominant measure of divergent thinking as it 
captures the major components of divergent thinking and creativity” (Sternberg, 2000). There are two major test 
forms, Verbal and Figural, and the Figural forma was used in this instance. It has a number of elements that are 
scored. “The test consists of six sections (Fluency, Flexibility, Elaboration, Originality, Abstractness of Titles, and 
Resistance to Closure) that ask for multiple written responses to illustrations and verbal prompts. Each section is 
timed, with responses measured within 5 to 10 minutes each” (Im, Hokanson, & Johnson, 2015). 


Effects of Lifestyle/Income on Creativity 


Torrance, in his research, found the TTCT was not-biased against or in favor of “disadvantaged” groups. 
“,. some of our most outstanding performers from the very beginning were children from definitely disadvantaged 
backgrounds” (Torrance, 1971). This will allow an accurate correlation between income level and creativity among 
school children. 

There are many factors that inhibit creative development that deal directly with being socioeconomically 
disadvantaged. “There are seven main reasons why students from low income households are more likely to struggle 
with classroom engagement” (Jensen, 2017). Low income families may lack the same opportunities that high 
income families are be able to offer their children. There are numerous small elements that are also missing in 
children's lives in poor families. “Low income students may also lack the resources that are essential to creative 
development. For example, pretend play and outdoor play in an individual’s childhood years is a critical component 
in developing higher levels of creativity” (W. Russ, 2012). “Creativity appears early in a life and it shows in the 
child’s play. Gradually it spreads to other areas of life.” (Mankar, Ugale, Rothe, 2011). Childhood play offers 
cognitive and affective attributes that aid in child creative development, but may not be as effective for children in 
low income families if they do not have access to toys, books, kid-friendly educational technology, outdoor 
equipment, or other such resources. 

There are other pragmatic reasons that disadvantage poor children. Children from socioeconomically 
disadvantaged households often have to start working earlier in life, compared with those with higher household 
incomes. They may lack the means of transportation to get to school, or they might be required to take care of 
siblings while parents are at work. Impacts from a more stressful life can include a lack of motivation, negative or 
disobedient behavior, or being distraction from academic and creative performance. Results could also include: 
health, vocabulary, effort, hope and the growth mindset, cognition, relationships, distress. These can all influence 
the deterioration of creativity (Votruba-Drzal, Miller, Coley, 2016). 

An additional developmental impact of poverty is the inability to participate in formal school-based 
activities such as a sports team or club which encourage a range of skills. Members learn how to be part of a 
cohesive unit and work together. They build relationships and learn about different personalities, different cultures, 
and social skills. When low income families can’t afford the time or money to include their children into these clubs 
or sports it slows down their creative development. 

A study by Mitchell (1975) revealed a number of insights that affect creative development and pertain to 
children from low income families. “One of them is that children frequently come from a home environment with 
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such a paucity of objects that the child’s conceptual information formation development is adversely affected” 
(p.46). He found having a concrete object in front of a child helps build identity similarities, differences and develop 
curiosity in order to gain an understanding for new situations and experiences. 

A child develops curiosity by having things to be curious about. A lack of curiosity affects both 
motivational patterns and the development of creative behavior. Generally, low income families may be unable to 
provide their children with a wide variety of experiences, the ability to provide a range of toys or travel, a moderate 
stress environment, or time for personal development. 

It is easy to see there are many elements that students in low income families may lack which detract from 
having the opportunity to build creativity or be creative. Experiences with objects and experiences from visiting 
new places can aid in a child’s ability to create new and original ideas; these are all economic opportunities, both 
immediately and developmentally. 


FRP as an Indicator for Socioeconomic Status 


“The Free or Reduced Price meals program (FRP) is a federally-assisted meal program that operates within 
public and nonprofit private schools to provide low-cost or free lunches to students on a daily basis” (Hinrichs, 
2010). In the United States, qualification for the FRP program is generally accepted as an indicator of limited family 
income. Identification of students participating in the FRP was included in the demographic data received from the 
district research department. 

Qualification in the RFP program varies by state: “To qualify for the FRP in Minnesota, annual family 
income for a four person family must fall under $44,955” (welfareinfo.org, 2017). This income level is about 190% 
of the state poverty level: “In 2013, the poverty level was $23,500” (welfareinfo.org, 2017). This threshold remains 
significantly below the median statewide household income of $71,000 which would represent the lower end of 
income ranges in Minnesota. Therefore, the FRP income threshold correlates with the “low income” level of 
households in Minnesota and can reasonably be used as a surrogate parameter to identify students living in low 
income households that in turn may hinder the development of creativity. By using the FRP as our surrogate for 
measuring low income, we can separate students into a group that includes low income students (FRP group), and a 
group to represent a standard population (Non-FRP group). 


Motivation and Significance 


Research is continuously searching for positive and innovative ways to help individuals reach their highest 
potential. This study is motivated by the need to improve programs to help students who come from 
socioeconomically disadvantaged backgrounds in order to give them the same opportunities in creative development 
as have their better-off counterparts. By finding a link between socioeconomic status and creativity, we will have a 
better understanding how to assist students in reaching their highest potential. Our hypothesis is that students who 
are enrolled in the FRP will score significantly lower than students who are not enrolled in the program as measured 
by creativity testing of middle school students ages 13 to 14 in Minnesota. 

This research is significant in that an individual at a lower socioeconomic class will have lower skills in 
creativity, and that creativity is tied to lifetime achievement. “Results suggest that just under half of the variance in 
adult creative achievement is explained by divergent thinking test scores, with the contribution of divergent thinking 
being more than 3 times that of intelligence quotients” (Plucker, 1997). 

If this study can provide a connection between socioeconomic status and creativity then programs can be better 
justified in efforts to help disadvantaged individuals be access more resources and assistance. 


Methods 


For this study, data was formally collected in 2013 by the school district research department. 995 8" graders 
from 6 different middle schools in a Minnesota district were administered the Visual format of the Torrance Test of 
Creative Thinking (TTCT). The 8" graders’ ages ranged from 13 to 14 years old and are a roughly equal mix of 
male and female students. 

Within each school, half of the homerooms were used, and were chosen at random. Tests were administered 
over three different class meetings by the home-room teachers, with one of the three prompts presented each session. 
8th graders as a group were used they were also annually tested using the Minnesota Comprehensive Achievement 
Test (MCAT) and their academic performance could be compared. While other observations have been made about 
this population, the focus on this study is on the impact of lower income on creativity. 
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The students were separated into 2 groups based on their family’s income. One group included students enrolled 
in the Free or Reduced Price meals program (FRP) and the 2nd group is students who were not enrolled in the 
program (Non-FRP). 

Demographic and other academic information was provided by the school district’s research department. They 
provided data including GPA, MCAT scores, discipline problems, and participation in the Free and Reduced Price 
meals program. They were also responsible to remove individual identification from the data. 

The sample set can be considered representational of 8" grade students within the State of Minnesota because 
the district contains a variety of economic conditions and urban, rural, and suburban lifestyles. All schools in the 
study were public institutions. 


Materials 


The Torrance Test of Creative Thinking (TTCT) is the most common and accurate tool used in measuring 
creativity (Kim, 2006). The TTCT’s purpose is to quantitatively assess the creativity of its test subjects. Individuals 
are asked to complete a variety of visual tasks, such as drawing pictures and adding titles their drawings. Completed 
tests were collected from all schools and sent to STS Testing, the publisher for scoring. The publisher's scoring 
interrater reliability is better than .90 (Kim, 2006). 

After the test was completed, scores were broken down into six components of creative thinking: Fluency, 
Flexibility, Resistance to Premature Closure, Originality, Elaboration, and Abstractness of Titles. We then 
calculated the mean, median, and standard deviation from each group within the 6 components of the TTCT. These 
calculations indicated an overall difference in performance between the FRP group and the Non-FRP group. 


Statistical Analysis 


Each of the sub-sections of the Figural Torrance Test were analyzed separately. The means calculated were 
separated into the two variable groups, FRP and Non-FRP. The Non-FRP group consistently performed better on 
each metric in terms of a higher mean and median. Simple T-Tests were conducted comparing for each of the six 
scoring metrics between the FRP group and the Non-FRP group to assess if there was significance (<0.05) for each 
of the six scoring metrics. Results are presented below in Table | as well as through Figures 1-6. This is followed by 
a brief commentary on the findings. 


Results 


The study found students who are enrolled in the Free and Reduced Lunch plan score significantly lower on 
the TTCT than students who are not enrolled in the Free and Reduced Price mean program. The table below shows 
the mean and standard deviation from the data set for each metric. Table 1 shows the results from students who took 
that TTCT that are not enrolled in the Free and Reduced Lunch Plan. Table 2 shows results from students who took 
the TTCT that are enrolled in the Free and Reduced Lunch Program. Table three shows the T-Test Results from 
comparing students enrolled in the FRP compared to Students not enrolled in the FRP. Graphs 1-6 below show the 
standard deviation of the results from each of the 6 scoring metrics of the TTCT. 
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Fluency Flexibility Originality Elaboration Titles Closure 


Non - FRP 24.35 10.23 17.07 7.98 6.38 13.55 
Mean 
Non - FRP 9.22 4.12 7.00 2.73 4.00 3.85 
STDEV 
FRP Mean 23.28 15.75 13.10 


FRP & Non- 0.098 0.041* 0.006* 0.000* 0.022* 0.103 
FRP 


* Significant at p<.05 
Table 1: Means, Standard deviations, and T-Test comparisons of FRP and Non-FRP groups 


Analysis of the TTCT Test results (by t-test) of the two groups for Fluency and the Resistance to Closure 
metrics found no significant statistical difference. In both instances, the non-FRP scored higher than the FRP group. 


On the other hand, there is a significant statistical difference ( at 0.05) between the groups on the 
Flexibility, Elaboration, Originality, and Abstractness of Titles scoring metrics. Figures 3 - 6 illustrating the findings 
are pictured below. These graphs represent the distribution of test results (horizontal) and test subjects (vertical) for 
both groups. 

Discussion 


It should be noted that in all areas, the means of the FRP subgroup are lower than the means of the Non- 
FRP group. The standard deviation of the FRP group is much higher, indicating a broader range of scores and 
creativity skill. While not significant, it may indicate a diverse difference in capability between the two groups. 

The t-test results for Fluency were p= 0.098>0.05; for Resistance to Closure, p=0.103>0.05. Fluency is the 
ability to generate a large number of answers to a given prompt. Resistance to Closure is the ability to accept 
ambiguity and not complete a figure. This could be due to random error within the data set, or an actual closer level 
of skill between the two groups. While the difference in means is not significant between the two groups, the Non- 
FRP did perform at a higher level. This result does not indicate any significance in relation to this comparison of 
socioeconomic status and creativity within our study. 

On the other hand, the results of the t-tests for the Flexibility, Originality, Elaboration, and Abstract of 
Titles were all statistically significant. Flexibility is the ability to generate different types of ideas; it was significant 
in our study a p=0.041<0.05. A graph of distribution is shown in Figure 1. 
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Figure 1: Flexibility Metric Results Distribution 


Originality is the capability to create ideas which are unusual or rare, and it generally benefits from 
exposure to a broad range of experiences, including travel, diverse educational experiences, and involvement in 
cultural events. It was significant at p=0.006<0.05. Originality is generally connected to exposure to divergent and 
different ideas. The distribution of results is shown in Figure 2. 
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Figure 2: Originality Metric Results Distribution 
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Figure 3: Elaboration Metric Results Distribution 


Elaboration is the ability to add details to any single idea. Statistical significance was found with 
p=0.00<0.05. See Figure 3. 
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Figure 4: Abstractness of Titles Results Distribution 


Abstraction of Titles reflects the ability synthesize different ideas and present these ideas in the labeling of 
images. This metric was significant at p =0.022<0.05 and is shown in Figure 4. 
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These results show there appears to be a relationship between socioeconomic status and creative testing 
scores. Based on the data, comparisons of four of the six scoring metrics were statistically significant (>0.05), and 
therefore the hypothesis is accepted. 


Within this research, decreased creativity development is tied to low socio-economic status. This and other 
comparable research would argue a need for programs to benefit students from low income families. How low- 
income parents can be made aware and address the importance of creative development remains open to discussion. 


Limitations and Suggestions for Future Research 


One of the pitfalls of a cross-sectional study is not being ale to measure an exact representation of a 
population. Using FRP as a proxy for low income status is not a completely accurate measurement tool, but serves 
as the best method to date for measuring income. Thus, one of the limitations of this research is the difficulty of 
measuring families who do not enroll their child in the Free and Reduced Lunch program when their socioeconomic 
status qualifies. This may be due to a variety of reasons, one being parents who do not want to be associated with the 
stigma of low income, or do not want to advertise their socioeconomic status. Parents and students may also be 
unaware of the program being available, or forget to enroll their child. On the other hand, families may “cheat the 
system” and enroll their child in the Free and Reduced Lunch program when they do not qualify. These were 
accepted as minor errors which did not affect the results of the study. 

We offer several additional suggestions for future research. Because our results indicate a connection 
between low income and low creativity scores, schools can justify providing more resources and diverse experiences 
to low income students to provide more opportunity for improving their creativity and strive for excellence. E. Paul 
Torrance has stressed that to provide disadvantaged children a chance to develop their creativity, school and 
community programs have to respect the creative positives of disadvantaged children and build upon them 
(Torrance, 1971). 

In addition, comparisons of each of the six scoring metrics of the Torrance Test of Creative Thinking can 
provide more of an in-depth understanding of what aspects students in various socioeconomic classes score higher 
or lower in, and what components of the metrics are the most effective in identifying areas of creativity. Researchers 
may also want to identify cognitive abilities that hinder or enhance the development of creative problem-solving 
skills within students in low income families (Im, Hokanson, Johnson, 2015). Another suggestion would be to focus 
the direction of studying creative development to socioeconomic status, or even poverty, because it is less common. 
Since the majority of research conducted for creative development is focused on effects of ethnicity or gender on 
creativity, further research focused on the relationship between socioeconomic status and creativity would be 
beneficial. 

In conclusion, it is critical that programs and resources are available for schooling systems to help students 
of low-income families to have equal opportunities as students from middle to high income families to enhance their 
creative development. It is also critical that further research is conducted to compare the effects of low income to 
creative development in other populations and cultures. 
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Abstract 


Educational technologist performs as a catalyzer in supporting active and effective learning in an educational 
system. In Thailand, Educational technologists’ roles are delineated in the Chapter IX of 1999 National Education 
Act 1999 that were practically media profession who analyze, design, develop, and deliver media in a teaching and 
learning system such as radio, television, text book, and printed media, as well as all types of communication 
technology, while using research as a tool in the process, and practical professional development as well. None of the 
study has been done to solid instructional technology standards and its proficiency in Thailand. To strengthen the field 
of instructional technology, the study is aimed to research standards and competency for Thai instructional 
technologists. 

The study was employed by systematic reviews of current research and existing competency standards of 
educational technology after year 2010, coupled with an in-depth interview with experts in educational technology 
filed. The major standards of revision were retrieved from mainly organizations in the United States including: 
Association for Educational Communications and Technology: (AECT standards, 2012), the International Society for 
Technology in Education (ISTE standards, 2016), and including such as ISTE Standards for students (2016), and 
including major standards modified on the basis of ISTE standards namely the Washington State K-12 Educational 
Technology Standards, and Michigan Educational Technology Standards. 

The study was found the proposed educational technology standards including six areas: |.profound 
knowledge in technology and its adaptation to education, 2. system design, 3.professional development, 4.learning 
environment, 5. research and ethics, 6.localized and globalization. The proposed standards should be 
further developed and will be a potential countrywide implementation. 


Keywords: educational technologists, educational technology standards, Thailand 


Introduction 


The rapidly changing digital world today has affected to all parts of human society, so we should adapt to 
these changes in order to survive sustainably. It can be said that the advance technologies affect to all career paths 
because people have to face to the challenge of changes as well as the educational technology profession. Educational 
technologists are the one of the top professions that they has the duty to develop education. Their responsibilities are 
to facilitate and support the educational system and instruction for the teachers to improve the students’ learning to be 
more effective. 

The literature review found that educational technology standards were not developed or improved by the 
main organizations of the educational technology profession. They are just the researches of graduated and doctoral 
students who develop the standards and competencies of educational technologists. Also, there are a few and most of 
them are over 10 years. Thus, it shows the lack of continuous development and awareness of the importance about 
profession which can affect to the role of educational technologists in the society. So, the standard developments of 
the educational technologists are not developed consistently. This problem has been the same as many educational 
technologists in many countries. Actually, educational technologists are very important in order to support the 
development of educational systems and lead them to the good ways. On the other hand, they rarely have role in the 
society. This is due to the lack of a clear professional structure (Fox and Summer, 2014), or it can be said that this is 
a concern for the overlap of educational technology roles (Browne and Beetham, 2010). This concern has occurred to 
the wide educational technologists. The research has reported that the challenges of 21‘ century technology educators 
were affected by the advances in technology. How will they guide their educational goals? And how they use the 
technologies to develop learning and teaching methods in order to improve the student knowledges and skills? (Mayes, 
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Natividad, and Spector, 2015). According to these problems, the study is aimed to research standards and 
competencies for Thai instructional technologists. 

The development of educational technology standards positively affects to the professionalism of educational 
technologists in improving the performance and strengthening the educational professional organizations. However, 
the development of such standards should be continued consistently. This will be beneficial for the development of 
Thailand's education, which directly affects to the learner's ability. 


Literature Review 
Educational Technology Standards in Other Countries 


For the development of international standards, the researcher has studied the standards of educational technology of 
AECT (2012), which has been defined five standards as follows: standard 1 content knowledge, standard 2 content 
pedagogy, standard 3 learning environments, standard 4 professional knowledge and skills, and standard 5 research. 
AECT is one of the world's leading educational technology providers, and Thailand is also one of the members of 
AECT. Therefore, a trend study of AECT technology standards is necessary. 

In addition, the researcher has studied the standards of the International Society for Technology in Education 
(ISET) which is a nonprofit organization. It focused on providing all educators with the ability to control technology 
to accelerate innovation in instruction and inspire learners to achieve high potential. It has developed standards for 
educational groups including educators, coaches, teachers, and learners as the following details. 

ISTE Standards for educators (2017) is a standard that encourages learners to have greater potential for 
learning. It also promotes peer collaboration and interaction with other educators in finding new approaches. The 
development of various areas to support the learning of the students to the highest efficiency. It consists of 7 standards, 
the first standard is a developer of continuous learning by technology, creation and also the participation in local and 
global learning network, the second standard is a leader who is visionary to impel advancement in developing digital 
learning and learning that is equal to the student's success, the third standard is a positive inspiration for learners in 
the digital world, the fourth standard is a collaborative effort with others using concepts and taking the technology 
and new digital resources to solve the problems and create a real-world learning experience, the fifth standard is a 
truly learning activity designer by using tools, digital technology in many real-world environments, the sixth standard 
is a facilitator of learning with the use of digital technology to support the development of learning outcomes according 
to creative standards, and the last standard is a knowledgeable and logical analyst who promotes alternative learning, 
designing and implementing multiple assessments to help learners achieve their learing goals. 

The part of the ISTE standard for teachers (2008) is defined in five aspects. 1) Teachers facilitate and inspire 
the students to learn and to think creatively, (2 teachers can design and develop learning experiences and assessments 
in the digital age, (3 be a professional teacher who can demonstrates competence, skill and expertise in innovation, 4) 
teachers promote and define responsibilities for digital citizenship, and 5) teachers participate in professional growth 
and leadership. The ISTE standards for teachers have been applied in the development of standards for students in 
order to improve their learning in various areas to increase their potential. 

Moreover, ISTE standard for student (2016) developed standards for learners by focusing on student-driven 
learning processes, consisting of seven standards 1) students use technology to demonstrate their potential to achieve 
their learning goals, 2) recognize liberty and rights, responsibility, opportunities of living, learning and working in 
the digital world, 3) value learning resources, especially digital tools, to be used in self-learning and others, 4) use a 
variety of technologies in the design process and solve problems to create new things, 5) develop and use problem- 
solving and understanding strategies with technology-assisted methods, 6) students can communicate creatively with 
digital tools and media, and 7) digital tools are used by learners to extend their perspectives and learning. It also works 
with others effectively in both local and global teams. 

However, the ISTE standards are the model used by states in the United States, which will be adapted to the 
context of their own state, for example the Washington State K-12 Educational Technology Standards (2018) have 
seven standards. Apart from that, Michigan adopted the ISTE Standards for Students as competencies for technology 
integration (MITECS) during the 2017-2018 school year, there are also Kentucky standards for teachers. Other 
countries have also adopted the ISTE standards, such as Pakistan. 
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Educational Technology Standards in Thailand 


The development of educational technology standards in Thailand has been developed, but is still very limited and 
lack of continuity. Educational technologists’ roles are delineated in the Chapter IX of 1999 National Education Act 
1999 That the importance of educational technology has been determined in seven areas as follows: Educational 
technologists are professionals in the media by analyzing, designing, developing and providing media services in 
teaching and learning systems such as radio and television, text book, and printed media, as well as all types of 
communication technology, while using research as a tool in the process, and practical professional development as 
well. 

The study or research on the development of educational technology standards is still low and lack of 
continuous development. However, there are also those who develop this. Thapanee (2003) has developed 
performance standard for educational technology in higher education institutions in Thailand consists of 14 standards 
and 84 indicators; 1) Instructional design and instructional development standard, 2) Instructional media design, 3) 
Educational technology training, 4)Research and development, 5) Internal and external assessment, 6) Educational 
technology diffusion, 7) Graphic media production, 8) Printed material production, 9) Video/television production, 
10) Audio material/radio production, 11) Computer media production, 12) Education technology consultation service, 
13) Instructional media and audio visual aids service, and 14) Planning and management. 

In addition, Wasan et al. (2006) developed the national standard for educational technology for the institute 
of graduate study. It consists of 9 standards, 34 indicators as follows: 1) the institutional leadership in educational 
technology, 2) the curricular structure of teacher education program, (3) the technology infrastructure of educational 
technology, 4) the technology innovations fostering the instruction in teacher education institution, 5) the personnel 
in educational technology, 6) the faculty members’ performance in educational technology, 7) the instructional models 
fostering the pre-service teacher students’ uses of educational technology, 8) the professional experiences, and 9) the 
pre-service teacher students’ performance in educational technology. 

From literary review by analyzing, synthesizing the research papers related to the standard, it is shown that 
Educational technology standards need to be developed urgently and should be continually developed to create 
professional standards for technologists to be strong and up-to-date with the rapidly changing digital technology world. 
For the benefit of education systems that affect learners’ learning and instructor effectiveness to maximum potential. 


Research Objectives 


The study is aimed to research standards and competencies for Thai instructional technologists. 
Research Methods 
Researchers have developed a study on the development of educational technology standards of each organization 
both in Thailand and abroad as follows. 

1. Study and analysis of educational technology standard documents for educational technologists as well as 
teachers and educational personnel of foreign agencies. 

2. Develop a draft standard for educational technology based on the review of research and educational 
technology standards from organizations in other countries. 

3. In-depth interviews with experts in the field of educational technology were conducted to gather 
information, opinions and expert advice on the drafting of educational technology standards. 


Research Results 


According to studies, it has been found that researchers can synthesize draft standards of educational technology for 
six standards, as follow. 

Standard 1 Profound knowledge in technology and its adaptation to education. There are three indicators 
include: 1.1) profound knowledge of both theoretical and practical education technologies, 1.2) profound knowledge 
about education, 1.3) knowledge of new technologies. 

Standard 2 System design. There are 2 indicators include, 2.1) design of teaching and learning system, 2.2) 
design of media system. 

Standard 3 Professional development. There are three indicators: 3.1) expertise in media production skills, 
3.2) leadership in new technologies applied in teaching, 3.3) be a consultant to develop learning by using advanced 
technology. 
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Standard 4 The learning environment. There are two indicators: 4.1) the development of a learning 
environment that uses technology in various ways, 4.2) the organization of a diverse learning environment based on 
real conditions. 

Standard 5 Research and ethics. There are two indicators: 5.1) Applied research in education and educational 
technology, 5.2) Research in development of media, 5.3) Ethics of Educational Technology and rese. 

Standard 6 Localized and globalization. There are two indicators: 6.1) the application of knowledge in the 
field of technology and the use of local networks, 6.2) the creation of a broader network to the global network. 


In terms of in-depth interviews of experts with interesting issues, as detailed below. 

Standard 1 Profound knowledge in technology and its adaptation to education. "Standards 1 should identify 
indicators of educational attainment in relation to teacher professional standards and student performance in Thailand 
as teachers and learners are key target groups for educational technology. Therefore, standards of educational 
technology need to be consistent with the standards and competencies of teachers and learners." (Experts 1, 4 and 5) 

Standard 2 Design System "should include theories of learning, psychological theory, perception theory, 
media characteristics in design." (Experts 3, 4 and 7) 

Standard 3 Professional development “Identify skills, expertise in thinking, creativity, use of all learning 
materials, and technology in advanced platforms.” (Experts 2, 5 and 6) 

Standard 4 Learning environment “The ability to utilize the atmosphere and suitability of the classroom and 
other learning facilities should be discussed.” (Experts 3, 4 and 7) 

Standard 5 Research and ethics "Standard 2 should focus on innovative research to be in line with the 
Thailand 4.0 policy and should include research ethics, progressive thinking, application and contemporary 
thinking.” (Experts 1, and 3) 

Standard 6 Localized and globalization “Identify the actual implementation. In accordance with the policy 
of Thailand 4.0, but consistent with the culture of culture in the context of Thai society.” 

(Experts 2, and 5) 

In addition, experts have proposed that additional standards or clear indications be added. 

1) Ladership standards should be added as most educational technologists play the role of a leader that is 
not as clear as it should be. Therefore, it should be defined as one standard that will improve the performance of 
educational technologists to a higher level. (Expert 2, 3 and 6) 

2) The standard of the role counselor or facilitator of educational technology should be identified. (Experts 
1, 3, 4 and 7) 


Discussion 


The standards developed by the researcher consist of six standards which 1, 3, 4 and 5 standards are consistent to the 
AECT (2012) standards, but differ in their metrics and focus. Standard 1 Content Knowledge has a different focus: 
Standard: 1, which develops profound knowledge in technology and its adaptation to education, focuses more on the 
knowledge of the educational system that involves teachers. Next, standard: 3 professional development, for example, 
defines an indicator that identifies media production skills, being a consultant, and a technology leader, which are not 
the same thing. Standard 4: the learning environment is focused on the environment where technology is used to learn 
and the learning environment is realistic. And standard 5: research and ethics, the emphasis is on the name of the 
standard, the code of ethics, which shows the ethics is important and the research has two indicators, namely, applied 
research and development research. 

While standard2: system design, conforming to standard 2 of ISTE for Teachers (2008) Some of the design, 
development, evaluation, and learning outcomes of learners. It does not mention the system. The standard 6: localized 
and globalization complies with ISTE standards for educators (2017) in Standard: 1 and Standard: 7 for students. 
However, based on expert interviews, some ideas need to be added to the standards that define the role of educational 
technology more clearly: the standards of leadership and the standards of counsel and facilitation. In accordance with 
the ISTE Standards for Teachers (2008) and ISTE for Educators (2017). 


Conclusion 
The standard of educational technology in Thailand consists of 6 standards; 1) profound knowledge in 
technology and its adaptation to education, 2) system design, 3) professional development, 4) learning environment, 


5) research and ethics, and 6) localized and globalization Based on interviews with experts, it was found that the 
experts gave more suggestions on adjusting standards for greater coverage. The researcher has to take this effect to 
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further study and make appropriate adjustments. The results of this study are consistent to the recommendations of 
the panel of experts to ensure that the educational technology standards are fulfilled before proceeding with the 
research. 
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Abstract 


The purpose of this study was to determine learner satisfaction towards techniques and strategies used to 
establish social presence in an online course. Qualitative research methodology was used. Study had two phases. In 
the first phase, an online course including techniques for establishing social presence was designed and developed. 
In the second phase, online course was offered to students. After the course, an open-ended question form was 
administered to students. Participants of the study were 22 senior undergraduate students at a public university in 
Turkey. Students took a course in a blended format. The online course was designed with techniques for establishing 
social presence. These techniques were course orientation videos, audio-visual meetings, providing frequent and 
detailed feedback, limiting class size, using sense of humor, using emoticons, addressing students by name, sharing 
personal stories and experiences, expressing agreement or disagreement, asking questions and inviting response and 
greetings. According to the findings, students were highly satisfied with the course which was designed and 
facilitated with social presence features. 


Keywords: Social presence, learner satisfaction, online course design 
Introduction 


Social presence is defined by Short, Williams and Christie (1976) as "the degree of salience of the other 
person in the interaction and the consequent salience of the interpersonal relationships". Social presence occurs if a 
person perceived as a “real person” in an online community (Garrison, Anderson & Archer, 2000; Gunawardena & 
Zittle, 1997). 


Learner satisfaction in online learning environments is important since it positively affects student 
achievement (Lowenthal & Dunlap, 2018). Social presence positively affects learner satisfaction (Gunawardena & 
Zittle, 1997; Hostetter & Busch, 2006; Johnson, Hornik & Salas, 2008; Swan & Shih, 2005) and learner 
achievement (Richardson & Swan, 2003). In addition, Oyarzun, Barreto and Conklin (2018) found that instructor 
social presence positively affects learner achievement. Thus, it can be said that student and instructor presence have 
positive effects on learner satisfaction and achievement. There is limited study in that topic and this should be tested. 
In this context, the purpose of this study is to determine learner satisfaction towards techniques and strategies used 
to establish social presence in an online course. 


There are some techniques and strategies for creating online presence in online courses. Lowenthal and 
Dunlap (2018) offered some techniques to establish social presence: 
e Introductions: Online courses should begin with introductions. For instance, course instructor and students 
can introduce themselves. 
e Orientations: Short orientation videos can be used at the beginning of the course. 
Personalized detailed feedback: Instructor can provide feedback individually in different environments. 
e Reconnecting: Instructors reconnect students with some activities to know their personality. With this 
activity, students can know each other’s personality, too. 
e Free-flowing, organic interactions: Twitter (Dunlap & Lowenthal, 2009) or Facebook groups (Izmirli, 
2017) can help for interaction. 
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Aargon (2003) stated some strategies to establish social presence: 
Developing welcome message like a video introducing instructor and course. 
Including student profiles in the course web page 

Using audio 

Limiting class size (Maximum 30 students) 

Using collaborative activities 

Contributing discussion boards by instructors 

Answering e-mails 

Providing frequent feedback 

Opening a conversation 

Sharing personal stories and experiences 

Using humor 

Using emoticons 

Addressing students by name 

Allowing students option for addressing the instructor 


Method 
Design and Participants 


Qualitative research methodology was used to determine learner satisfaction. Study had two phases. In the 
first phase, an online course including techniques for establishing social presence was designed and developed. In 
the second phase, online course was offered to students. After the course, an open-ended question form was 
administered to students. 


Participants of the study were 22 senior undergraduate students at a public university in Turkey. 13 of them 
were male and nine of them were female. Their ages ranged from 20 to 28. Students took a course in a blended 
format. The online course was designed with techniques for establishing social presence. 


Online Course Design and Development 


An online course was designed and developed. Online course had two main environments. First one was 
learning management system (LMS) which is Moodle in here. Second one was web conferencing environment 
which is Adobe Connect in here. In the course, course design and facilitation strategies were considered to establish 
social presence. These are as follows: 
e Course orientation video (Aargon, 2003; Lowenthal & Dunlap, 2018) 

Audio-visual meetings (Aargon, 2003; Gunawardena & Zittle, 1997). 

Providing frequent and detailed feedback (Aargon, 2003; Lowenthal & Dunlap, 2018). 

Limiting class size (Aargon, 2003) 

Using sense of humor (Aargon, 2003; Izmirli, 2007; Rourke, Anderson, Garrison & Archer, 1999; 

Richardson et al., 2015; Swan, 2003) 

e Using emoticons (Aargon, 2003; Izmirli, 2007; Rourke et al., 1999; Swan, 2003) 

e Addressing students by name (Aargon, 2003; Izmirli, 2017; Rourke et al., 1999; Richardson et al., 2015; 
Swan, 2003) 

e Sharing personal stories and experiences (Aargon, 2003) 

e Expressing agreement or disagreement (Izmirli, 2017; Rourke et al., 1999; Richardson et al., 2015; Swan, 
2003) 

e Asking questions and inviting response (Izmirli, 2017; Rourke et al., 1999; Richardson et al., 2015; Swan, 
2003) 

e =©Greetings (Izmirli, 2017; Rourke et al., 1999; Richardson et al., 2015; Swan, 2003) 


While some techniques and strategies helped to design the course, the others helped to facilitate the course 
in terms of creating social presence. 
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Data Collection 

An open-ended question form was prepared and used to collect data. In this form, students were asked for 
their satisfactions for each technique and strategy to establish social presence. Students stated their satisfactions for 
the design of LMS and web conferencing environment. In addition, students were asked for the overall course 
satisfaction with an open-ended and a 5 point likert-type question. 


Data Analysis 


In the analysis of qualitative data, descriptive analysis was used since there were predetermined codes 
which were techniques to establish social presence. In the analysis of quantitative data, mean was used. 


Findings and Discussion 


Learner satisfaction for online course design techniques and participant (course instructor and students) 
strategies to establish social presence are listed in predetermined themes. Themes are given in Table 1. 


Table 1. 

Themes for online course design techniques and participant strategies 

Themes Satisfied / Positive Not satisfied / 
Opinions (f) Negative opinions (f) 

Course orientation videos 21 - 

Audio-visual meetings 19 3 

Providing frequent and detailed feedback 20 1 

Limiting class size in live course 9 4 

Using sense of humor 18 5 

Using emoticons 17 1 

Addressing students by name 22 - 

Sharing personal stories and experiences 18 4 

Expressing agreement or disagreement 22 - 

Asking questions and inviting response 20 - 

Greetings 22 - 


Course Orientation Videos 


Instructor added a course orientation video in the beginning of the course in LMS. 21 students were 
satisfied with the course. Student 3 said that “It (adding course orientation video) created a sincere environment. 
Thanks to this, it was not hard to participate in the class”. Student 7 expressed that “It was positive because 
instructors’ introducing himself and course helped me to prepare for the course and to get used to the course”. 
Another student stated that “Since there was a sincere speech, I was not shy in the course”. Adding a course 
orientation video is very helpful for creating social presence and improving student satisfaction. This finding is 
parallel to Aargon’s (2003) and Lowenthal and Dunlap’s (2018) suggestion which is “introduction and orientation 
should be included in an online course to establish social presence.” 


Audio-Visual Meetings (Live Courses) 


In this study, audio-visual meetings were conducted via Adobe Connect. Most students had positive 
opinions for online live meetings. Student 2 said that “There was a sincere environment. I think every course should 
be given online”. Student 21 stated that “Live class provides everything equivalent to face-to-face class. It is like a 
real class. The only difference is that in the live class you are in front of the computer screen...”. This finding was 
parallel to the literature. Aargon (2003) stated that using audio is a strategy to establish social presence. But audio- 
visual communication triggers more intimacy than audio-only communication since it contains eye-contact 
(Gunawardena & Zittle, 1997). In addition to positive opinions, students stated technical problems like connecting to 
the live class. Student 5 expressed that “It (Live meeting) is perfect except technical problems”. 
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Providing frequent and Detailed Feedback 


Frequent and detailed feedback was provided to students in both LMS and live meeting. 20 students were 
satisfied with feedback provided. Student 9 said that “I felt that I was there (live class) and it is intimate like face-to- 
face class”. Similarly, Aargon (2003) and Lowenthal and Dunlap (2018) stated that providing frequent and detailed 
feedback enhance social presence. On the other hand, one student had negative opinion about feedback. He said that 
“Feedbacks are more effective in face-to-face classes. I think since live class was crowded, instructor cannot give 
feedback to everyone....” (Student 7). 


Limiting Class Size in Live Course 


Live class size was 22. While nine students were satisfied with class size in live course, four students were 
not satisfied. Student 13 expressed that “Class size in live class is good. But if the number of students increases, 
some students cannot engage in class actively. In that case, some students may drop out’. Student 10 who had 
negative opinion said that “Class size is high. Some students cannot engage in lesson”. According to this finding, it 
can be said that class size should be smaller than 22. In the literature there is no consensus on class size in an online 
course. Tomei (2006) found that ideal online class size is 12. Orellana (2006) found that optimal online class size is 
18.9 to have interaction. More studies are needed to determine ideal online class size. 


Using Sense of Humor 


Both instructor and student used sense of humor in both LMS and live meeting. Instructor used it 
intentionally to establish social presence. Most of the students were satisfied with the sense of humor in online 
course. One student said that “When sense of humor was used, I felt that I was in face-to-face class. I was at my 
home but it is like I am in the class...” (Student 16). This finding is parallel to the literature (e.g. Aargon, 2003; 
Izmirli, 2007; Rourke et al., 1999; Richardson et al., 2015; Swan, 2003). Some students stated their negative 
opinions about that topic. Student | said that “When it (sense of humor) was overused, we could not have effective 
lesson”. 


Using Emoticon 


Emoticons were used in LMS and live meeting by instructor and student. 17 students were satisfied with 
emoticons. Student 3 expressed that “Using emoticon was good because I felt that I was in the face-to-face class”. 
Another student said that “Using emoticon showed that people were sincere. There was not a serious environment 
thanks to emoticons. I felt happy” (Student 16). Just one student had negative opinion. She said that “I don’t think 
that it (using emoticon) is necessary” (Student 8). Like in the literature (e.g. Aargon, 2003; Izmirli, 2007; Rourke et 
al., 1999; Swan, 2003), using emoticon should be used to establish social presence. 


Addressing Students by Name 


Instructor addressed students by name in LMS and live meetings. All students were satisfied with being 
addressed by their names. Student 6 said that “I was satisfied that my instructor addressed me by my name. My 
interest towards course increased”. Student 17 expressed that “It (Being addressed by name) affected me positively 
because I understood that instructor knew me. One of the most important things for a student is being addressed by 
name. Because it is motivating”. This finding is parallel to the literature (e.g. Aargon, 2003; Izmirli, 2017; Rourke et 
al., 1999; Richardson et al., 2015; Swan, 2003). 


Sharing Personal Stories and Experiences 


Both instructor and students shared their personal stories and experiences in LMS and live meetings. While 
18 students have positive opinions, four students have negative opinions. Student 3 expressed her positive opinion 
with saying “I felt that it (sharing personal stories and experiences) created an environment as if I was in the 
classroom’’. Student 9 said that “... it was good since it helped to create a sincere environment”. According to this 
result, Aragon’s (2003) suggestion was confirmed. On the contrary, Student 8 said that “When it takes too much 
time, it affects the lesson negatively”. 
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Expressing Agreement or Disagreement 


In live courses, instructor and students expressed their agreements or disagreements by verbal expression or 
written text. All students were satisfied with expressing agreements and disagreements. Student 10 said that 
“Expressing agreement/disagreement positively affected the interaction between student and course”. Student 21 
expressed that “It provides that we participate in the lesson actively”. This finding is parallel to the literature 
(Izmirli, 2017; Rourke et al., 1999; Richardson et al., 2015; Swan, 2003) 


Asking Questions and Inviting Response 


Instructor asked questions and invited response in live meetings. 20 students stated that they were satisfied 
with being asked questions and being invited response. Student 7 said that “Being asked questions and being invited 
response affected me positively’. Besides student 16 expressed that “Instructor increased my motivation by asking 
me questions”. This finding is parallel to the literature (e.g. Izmirli, 2017; Rourke et al., 1999; Richardson et al., 
2015; Swan, 2003). 


Greetings 


Instructor greeted students with a friendly “hello” in each live meeting. All students had positive opinions 
about greetings. Student 8 said that “Greetings provides a more sincere environment”. Another student expressed 
that “I think it (greetings) creates a friendly class environment" (Student 13). This finding is parallel to the literature 
(e.g. Izmirli, 2017; Rourke et al., 1999; Richardson et al., 2015; Swan, 2003). 


Students’ Overall Satisfaction 


20 students were satisfied with the online course in general. Student 16 stated that “I was so satisfied in 
general. I want to do it (participation to online course) again. Many thanks for everything”. Student 18 said that “I 
was satisfied. Course was more comfortable, sincere and understandable”. Students were asked to answer a 
quantitative question (5 point likert-type) to measure their overall satisfaction. Students’ mean score was 4,59 which 
meant that students were highly satisfied with the course. 


Conclusion and Suggestions 


The purpose of this study was to determine student satisfaction towards techniques and strategies used to 
establish social presence in an online course. In the online course, some techniques and strategies were used to 
create social presence. These were course orientation videos, audio-visual meetings, providing frequent and detailed 
feedback, limiting class size, using sense of humor, using emoticons, addressing students by name, sharing personal 
stories and experiences, expressing agreement or disagreement, asking questions and inviting response and 
greetings. Students were highly satisfied with the course which was designed and facilitated with social presence 
features. Students had mostly positive opinions for the online course. Instructional designers and course instructors 
should consider social presence while designing an online course. 


This study has some limitations. Firstly, the study was conducted on online part of a blended course. 
Research studies should be conducted on fully online courses to find out learner satisfaction regarding social 
presence features. Secondly, participants of the study were senior undergraduate students. Studies with different 
participants can be conducted. Thirdly, the study examined student satisfaction. The effects of social presence 
features in online courses on achievement can be examined with experimental studies. 
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Abstract 


Video lecture is one of the most preferred learning resources in e- learning environments. Limited studies 
have investigated how video interaction data had an impact on learning performances. There is a gap in research 
pertaining to the relation between video interaction data and learning performance in real learning context. The 
purpose of this preliminary study is to explore how video interaction data of learners predict their learning 
performance. It has been founded that all variables reflecting video navigation behaviours of learners show a 
statistically significant relation to academic performance but not strong correlation. doSeek and viewing after 
feedback were significant in predicting the learning performances of the students. The regression model accounted for 
35.2% of the variance in the learning performance. Our preliminary study could make contribute to a better 
understanding the relationship between video interaction data and learning performance. 


Keywords: video based learning; video lecture; learning behaviours; learning performance 
Introduction 


In recent years, millions of learners have attended open courses offered by different universities and have 
participated in online communities for the courses along with thousands of international learners with time and place 
flexibility. In these courses, various electronic content and multimedia have been used any time from anywhere such 
as e-book, video lecture, content packages, pictures, animation, and simulation. Especially, video lectures are the most 
prominent learning resources since they have been widely used multimedia in open and distance learning 
environments (Giannakos, 2013). Online video lectures often provide learners to enrich learning experience with 
content combining visual and verbal and to allow using them repeatedly (Chen & Wu, 2015; Mayer, 2009). E-learners 
can interact with video lectures at their own pace whenever they want to view in open online learning environments. 
Interaction and pacing that improve the educational added value of video lectures are the main essential features of 
video lectures (Koko¢ & Altun, 2014; Sadik, 2015). 

In addition to video-based learning environments such as MOOCs and Khan Academy, video platforms like 
YouTube and Vimeo can be used individually or collaboratively for educational purposes (Kleftodimos & 
Evangelidis, 2016). Widely using video in learning has led to emerging a research field named as video-assisted 
learning and video-based learning. In this study, it has been preferred to use video-based learning as a concept. 
Numerous studies in video-based learning field indicated that using video lectures in the learning process increased 
learners’ involvement, course satisfaction, motivation and interest in learning (Giannakos, Chorianopoulos, & 
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Chrisochoides, 2015; Donkor, 2011; Hsin & Cigas, 2011; Kurtz, Tsimerman, & Steiner-Lavi, 2011; Traphagan, 
Kusera, & Kishi, 2010) and the students regarded video lectures as useful and enjoyment (Boateng et al., 2016; Fee & 
Budde-Sung, 2014). On the other hand, a number of studies claim that using video lectures lead to absenteeism and 
procrastination (Griffin, Mitchell, & Thompson, 2009; Traphagan, Kusera, & Kishi, 2010) and had no significant 
effect on attendance and academic performance of learners (Leadbeater, Shuttleworthb, Couperthwaitec, & 
Nightingale, 2013; Wieling & Hofman, 2010). These different results show that there was mixed evidence about the 
efficacy of video lectures on learning outcomes (Bos, 2016). Thus, new studies should be made to investigate the 
efficacy of video lectures in learning considering learning outcomes and new data sources from learners. 

Together with the importance of studies on using video lectures, this study may shed light on understanding 
the learning process with video lectures by analyzing video interaction data of learners. During viewing video 
lectures, the interaction between learner and video lectures occur due to the progress bar on the video interface. 
Learners interact with video lectures performing various actions such as playing, pausing, resuming, backward and 
forward jumps using the progress bar (Kleftodimos & Evangelidis, 2016). In this context, video learning analytics 
studies provide the opportunity to researchers deeper understanding interaction learners and video lectures. It is clear 
that how students learn via video lectures is still a critical question to gain insight into the efficacy of video lecture in 
learning (Giannakos, Chorianopoulos, & Chrisochoides, 2015). Thus, video learning analytics may contribute to 
answering the important question based on interaction data of learners as the objective data source. 

As aforementioned, a large number of researchers have studied using video lectures for learning context 
based on self-report data in the last decade. A review study on video lectures showed that mixed results revealed 
regarding the impact of video lectures on learning outcomes in the literature (O’Callaghan, Neumann, Jones, & Creed, 
2017). 

Additionally, limited studies have investigated how video interaction data or video navigation behavior of 
learners impact on learning performance. There is a gap in research pertaining to the relation between video 
interaction data and learning performance in a real learning context. The purpose of this preliminary study is to 
explore how video interaction data of learners predict their learning performance. To do this, the correlation between 
video interaction data and learning performance has been calculated at first. Then, it has been explored which video 
interaction data predicts the learning performance of learners. This study as a part of our ongoing research aims to 
answer the following research questions: 

Research Question — 1: What is the relationship between video interaction data and learning performance? 
Research Question — 2: Which video interaction data predicts the learning performance of learners? 


Method 


The participants in this study were 101 undergraduate students enrolled in an undergraduate course recruited 
from a major state university in Turkey in 2016; 48 were female, and 53 were male among the volunteer participants. 
No participants had prior knowledge of the topic presented in the videos. 

The Computer Networks and Communication course delivered online, in this study. The study took five 
weeks. The goal of the course was to understand conceptional and applied knowledge of OSI Model, TCP-IP, IPv4, 
IPv6. The course adopted an online learning context included 10 video lectures developed by the instructor during the 
study procedure. The students accessed the video lectures weekly via a video-based learning environment. The video 
lectures were designed associated with two video lecture types including talking head (half of all) and picture-in- 
picture. Figure | illustrates each example of the video lectures with two types. 

The length of video lectures is ranging from six to 12 minutes. The students viewed the video lectures and 
participated in a quiz consisting of 12 questions relevant to the content of the video lectures, weekly. After 
completing the quiz, feedback was given to the students with information about their false answers in the quizzes. 
The students were directed to the related video lecture with feedback. For an example, when a student did not 
answer correctly the question about the function of the layer 3 in OSI Model, feedback directed him to view at the 
third minute seventh second of the relevant video lecture. That was not a mandatory feedback and student can view 
the video lecture given in the feedback if she wants. 

The data sources of the study consisted of the video interaction data and learning performance of the students 
as provided by the exam results. The learning performance was evaluated by using the results of the second mid-term 
exam consisting of 15 multiple-choice questions and 10 open-ended questions. The video interaction data gathered 
from log data reflecting video viewing behavior of the students. Sample screenshots have been presented in Figure 1. 
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Figure 1. Illustrations of video lecture types 


A detailed description of events indicating each video interaction data is provided in the Table 1. 


Table 1. Description of events in video lectures 


Event Type Event Description 

Viewing at once Completely watching a video lecture at once 

doSeek Dragging a video lecture from one time point to another time point 
doPlayPause Clicking the play and pause button 

Total Viewing Time Total time of watching all video lectures 

Viewing After Feedback Playing the relevant video lecture shortly after received feedback 


Results 
Correlation Analysis 
The results of the correlation analyses of the study variables are presented in Table 2. The correlation analysis was 
conducted prior to multiple regression analysis to find out correlations among the independent (video navigation 


behaviors) and the dependent variable (academic performance) (See Table 2). 


Table 2. Correlation analysis (n=101) 


Variables l 2 3 4 5 6 
Learning Performance 1.00 

Viewing at once 378 ~— 1.00 

doSeek 400 541 1.00 

doPlayPause 487 = 718 ~~ 459 1.00 

Total Viewing Time 389 = =6.640 ~—.348 914 1.00 
Viewing After Feedback 556 = 497 = .290 -—.672_~——«587 ~—s‘1..00 


The correlations between video navigation behaviour of the students and and their learning performance are 
presented in Table 2. It has been founded that the all variables reflecting video navigation behaviours of learners show 
a Statistically significant relation to academic performance but not strong correlation. These positive correlations 
suggest that the more frequently a student engaged with the videos the higher they tended to obtain. 
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Multiple Regression Model 


To address the second research question, the four variables indicating video interaction data were selected as 
predictors, and a multiple regression analysis was conducted to examine the predictability of the learning 
performances of the students. 

A series of tests were performed to examine the assumptions that may affect its reliability before performing 
the multiple regression analysis. The tolerance values were above 0.10 for all the variables confirming the absence of 
multicollinearity and also the VIF values were below 10. The Durbin-Watson statistic verified the absence of 
autocorrelation (Durbin-Watson = 1.67). 

As presented in Table 3, doSeek (B=.628, p=.011) and viewing after feedback (B =.335, p=.003) were 
significant in predicting the learning performances of the students. The regression model accounted for 35% of the 
variance in the learning performance (R7=.35, F (5,.95) = 11.84, p< 0.001). 

Table 3. Results of multiple regression analysis 
Academic Performance 


Predictors B SE B R? 
Constant 33.71 2.989 - 

Viewing at once -.105 314 -.039 

doSeek .628 241 .251* 352 
doPlayPause .009 .007 197 , 
Total Viewing Time -.015 054 -.035 

Viewing After Feedback 335 .109 S1Iz* 


**® yn <.01. * p<.05 


The remaining variables, viewing at once, doPlayPause, and total viewing time did not make significant predictions 
to the learning performances of the students. 


Discussion and Conclusion 


Recent studies have suggested that using video lecture in education can enhance learning outcomes and 
support the learning process with flexibility and pacing. For all that, the studies indicate that the impact of using video 
lectures on learning performance is not clear. To the best of our knowledge, currently, there are limited studies about 
this subject using video interaction data. In this preliminary study, it has been aimed to explore which video 
interaction data predict learning performance of learners. 

The results of the study indicate that two indicators from the video interaction data significantly predicted the 
learning performance. However, watching the video lecture at once, clicking the play and pause button, and total 
viewing time was not significant. The regression model with two variables explained 35.2% of the variance in the 
learning performances. These findings support those of previous research (Ozan & Ozarslan, 2016), which has 
emphasized that video lecture viewing behaviors of learners effect on exam score. 

One interesting finding is that watching the video lecture depending on received feedback predicted the 
learning performance. This result may be explained by the fact that quiz feedback could increase learners’ 
engagement in the video-based learning environment and might lead to goal-oriented viewing behavior that provides 
better learning with video lectures. Another important result was that dragging the videos from one-time point to 
another time point predicted the learning performance significantly. This result, while preliminary, suggests that 
seeking behavior may provide learner to gain a deeper understanding of course content in video lectures (Bos, 2016). 

The study is a preliminary research that was limited in terms of one specific course with one group of 
students and not considering learners’ characteristic. Further studies, which take the more video interaction data types 
and individual differences of learners into account, will need to be undertaken. 
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Abstract 

Interaction is one of the most important components of open and distance learning. According to Moore, 
who proposed one of the keystones on interaction types, there are three basic types of interaction: learner-teacher, 
learner-content, and learner-learner. From these interaction types, learner-content interaction without doubt can be 
identified as the most fundamental one on which all education is based. Using the learner-content interaction 
activities in course materials, Anadolu University, by its Open Education system, tries to involve learners in deep 
and meaningful learning practices. Considering the lack of studies adopting this approach, as well as its being a 
study on the use of e-learning materials in Open Education system, this research holds a big value in open and 
distance learning literature. In this respect, the present study aimed to investigate a) which learner-content 
interaction activities included in interactive courses are the most effective in learners’ achievement of verbal 
information learning outcomes and b) to what extent distance learners are satisfied with these learner-content 
interaction activities. 

For this study, a quasi-experimental research design was adopted. The 120 participants were divided into 6 
groups randomly. While 5 of these groups received different learner-content interaction activities as a part of the 
experiment, the other group served as the control group. The data were collected mainly through two instruments: 
pre-test and post-test. In addition to those tests, learners’ perceived learning was assessed with an item at the end of 
the program. The data collected from pre-test and post-test were analyzed by ANOVA, and in the light of the 
findings of this approximately 24-month study, suggestions for further design of e-learning materials within the 
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context of learner-content interaction activities will be provided at the conference. The current study is planned to be 
an antecedent for the following studies that will examine the effects of activities on other learning domains. 


Introduction 


There has been a long history of interaction in any educational settings (e.g., Dewey, 1938; Vygotsky, 1978), 
yet a relatively new one in distance education (e.g., Anderson & Garrison, 1998; Holmberg, 1983; Moore, 1989; 
Wagner, 1994). In its earlier times, interaction in distance education was defined by adopting two different approaches; 
a merely humanistic one as “in a restrictive manner to cover only those activities where the students is in two-way 
contact with another person (or persons)” (Daniel & Marquis, 1988, p.339), and a more mechanic one as “reciprocal 
events that require at least two objects and two actions” (Wagner, 1994, p. 8). In an attempt to provide more precise 
and agreed upon sub meanings for interaction, Moore (1989) drew attention on three types of interaction (learner- 
learner, learner-content, and learner-instructor). Anderson and Garrison (1998) extended the discussions in distance 
education literature on these three major types of interaction to the other three types of interaction (instructor- 
instructor, instructor-content, and content-content). However, nowadays the relatively limited understanding of 
interaction once accepted in distance education have been replaced with more dynamic and active forms of interaction 
enhanced with a wide scope of strategies and the latest implications in learning environments including simulations, 
games, hyperlinks, virtual worlds, discussion boards (Fuller, Kuhne, & Frey, 2011), semantic web, social media, and 
massive open online courses (MOOCs). 

Interaction serves many important purposes in distance education transactions. Mason (1994) has listed 
benefits of interaction at the affective level as increasing learner motivation and interest in the content; fostering 
learning in deep; and encouraging critical thinking. Moreover, some studies have shown that high levels of interaction 
have an effect on increased learner and teacher satisfaction (Keeler, 2006; Kuo, 2014; Su, Bonk, Magjuka, Liu, & 
Lee, 2005), and motivation (Mahle, 2007). Last, learner-content interaction has been suggested as a critical component 
specifically in distance education settings (Anderson, 2003). Zimmerman (2012) has supported this by pointing out 
the importance of the interaction with the course content, that is learner-content interaction, as a contributing factor 
for the achievement of learning outcomes and course completion. 

Interaction, in its all forms, can be perceived as an effective way to promote distance education (Su et al., 
2005). Therefore, this paper attempts to provide an insight into the theoretical frameworks, definitions, types as well 
as classifications of interaction in distance education contexts, on which there has been a continuous debate for years, 
yet no compromise at all. 

Implementing a variety of interaction activities, most distance education institutions try to engage learners 
in deep understanding by involving them in meaningful learning practices during their teaching processes. It is a fact 
that in the medium of distance education, activities, dimensions, functions, and the other components of online 
learning including the concept of interaction must be used distinctively from traditional face-to-face education. 
Especially, during the online learning material design and production processes, identifying appropriate activities 
within the context of interaction types holds a big importance. As can be seen in Table 1, Chou (2003) builds a 
framework by classifying interactive functions in online learning with respect to different interaction types. 


Table 1. The Framework for Interaction Types and Interactive Functions in Online Learning. 
Types of Interaction Interactive Functions in Online Learning 


Fixed-frame (menu) design 
Online registration 

Grade status tracking 
Assignment completion tracking 
Leamer — Interface Keyword search 

Software downloading 

Site map 

Database search 

Online problem diagnostics 
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Frequently-asked-questions (FAQ) 

Links to related educational sites 

Links to related learning materials 
Multimedia presentation (text, graphics, animation, audio etc) 
User guidance on system 

On-line quiz for self-evaluation 

Push media 

On-line help on content 

Learner contributing to learning materials 
Individualized learning database 
Individualized instruction 

Difficult Individualized test/quiz 

Study guidance 

Jokes 

Sweepstakes 

Educational games 

Email to instructors 

Email to Web master 

Bulletin board systems (BBSs) 

Learner — Instructor Chatrooms 

Comments on the sites, course, instructor, etc. 
Online survey 

Online voting 

Email to other learners 

Bulletin board systems (BBSs) 

Learner — Learner Chatrooms 

Class roster 


Learner — Content 


Table 1. The Framework for Interaction Types and Interactive Functions in Online Learning. Adapted from Chou, 
C. (2003). Interactivity and interactive functions in web-based learning systems : a technical framework for 
designers. British Journal of Educational Technology, 34(3), 265-279. 


From these interaction types shown in Table 1, learner-content interaction without doubt can be identified 
as the most fundamental one on which all online learning is based (Vrasidas, 2000; Anderson, 2003). In other words, 
efficacy, efficiency and attraction of distance education systems can be achieved by the practice of efficient learner- 
content interaction through appropriate activities. 


Apart from interaction types, there are also some other classifications of interaction such as interaction 
taxonomy (Schwier & Misanchuk, 1993), interaction levels (Sims, 1997), categories of interaction (Stouppe, 1998), 
and types of content interaction (Shank, 2003) (See Table 2). 


Table 2. Classifications of Interaction 

Taxonomy of Levels of Categories of Learner- | Types of Content 

Interactivity Interactivity Content Interaction Interaction 

(Schwier & (Sims, 1997) (Stouppe, 1998) (Shank, 2003) 

Misanchuk, 1993) 

Levels Levels Enriching Multiple choice quizzes 

e Reactive Object interactions True/false quizzes 

e Proactive Linear © pop-ups Click on object or text to 
Hierarchical e hot-words reveal more information 
Support e links (glossary — explanations) 
Update e forward and back Hypertext links to other 
Construct buttons pages inside the course or 
Reflective program 
Simulation Supportive 
Hyperlinked | interactions 


e Mutual 


Functions 
Confirmation 
Pacing 
Navigation 
Inquiry 
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e Elaboration 


Transactions 
Keyboard 
Touch Panel 
Pointing 
Device 
Voice 


Non- 
Immersive 
Contextual 
Immersive 
Virtual 


zoom functions 
moveable ruler 
bars 

calculators 
orientation/ 
perspective 
controls 

search and query 
functions 


Conveyance 
interactions 
@ questions 


simulations 
games 

what-if activities 
process decision 
points 


Hypertext links to resources 
outside the course or 
program 

Tutorials (step-by-step) 
Drag and drop 
Navigational choices 
(choice of path and 
sequence of information) 
Application simulations 
(software demos or try-it 
simulation) 

Process simulations 
(realistic case studies — 
immersion exercises) 

Fill in the blank 
Self-reflection questions 
(Why do you think that...? 
Have you considered...?) 


Games 
Constructive Offline/field work (ie. watch 
interactions task, try program, offline 
building mental labs) 
maps Note taking/journals 
knowledge trees Simulated people (ask the 
organization expert) 
charts 


All in all, it is essential to decide which interaction type/s to use on the basis of learning environments, 
objectives, and learners’ needs and interests. In an online learning setting in which the intended learning group is 
composed of independent learners, it would be wise to foster more learner-content interaction on behalf of supporting 
and enriching content specific interaction. 


Methodology 


Although the subject and words are the same in eLearning content, different interaction activities are designed 
for each experimental group. Since the last group is a control group, only subject and vocabulary is presented without 
any interaction effectiveness. Groups and activities are summarized in Table 1. 

Table 1. Groups and Activities 


Group 1 Game (Puzzle) 

Group 2 Fill in the blank 

Group 3 Ranking 

Group 4 Pairing 

Group 5 True/false quizzes 

Group 6 No interaction (Control Group) 


Participants of the study were selected by convenience sampling method and then randomly assigned to the 
control and experimental groups. The data were collected with 2 basic tools, pre-test and post-test. Pre-test is presented 
as 17 questions (ANNEX-1). Post-test is presented as two parts. The first part consists of 17 questions English test 
(ANNEX-2) and the second part consists of gender, age, perceived learning, and satisfaction (Annex-3). In addition 
to this section, they were also asked whether they had previously taken English courses in the Open Education System, 
whether they had entered the system and whether they used tablets. 
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In addition to the information collected through pre-test, post-test and questionnaire, e-learning materials 
offered by HTMLS5 and SCORM have shown the answers to what the students answered at the event, how many points 
they got and how much time they spent. 

Data of 120 students collected in 6 different groups were analyzed by SPSS. The first 5 groups were the 
experimental group and the 6th group was the control group. To answer the question, “How effective are learner- 
content interaction activities in learning learners' verbal knowledge from Gagne's learning outcomes?”, the control 
group and all other groups were examined with ANCOVA analysis by using pre-test, post-test and survey data. 


FINDINGS AND COMMENTS 
In the study with 120 participants, 43% of the participants were female and 57% were male. 


When the correct answers given to the pre-test where the response rate is 99.1%, the result in Chart | is revealed: 


On Test Dogru Cevap Sayisi Dagilimt 


i Z 3 4 5 6 7 8 9 10 11 12 13 14 is 16 i7 


Chart 1. Status of correct answers according to pre-test responses 


Accordingly, it was observed that all participants gave at least 3 correct answers. The total number of participants is 
1. The response rate is 99.3% and the correct answers given to the post test are given in Chart 2. 
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Son Test Dogru Cevap Says! Dagilimi 


Chart 2. Status of correct answers in post test answers 


The correct response rate given in the pre-test was 41.5% (846 correct answers in total), the correct 
response rate was 52.2% (1064 correct answers in total). 


Data of 120 students collected in 6 different groups were analyzed by SPSS. The first 5 groups were the 
experimental group and the 6th group was the control group. The results of the analyzes performed within the scope 
of the question, ”How effective are learner-content interaction activities to learners' verbal knowledge from Gagne's 
learning outcomes?”, are given in Table 2, Table 3, Table 4, Table 5, and Table 6. 
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Table 2. Group | (Game) and Group 6 (Control Group) 


Source of | Sum of Degree of Squares F Partial Eta- 
Variance Squares | Freedom Mean Square 
_ ial | | 

_ 


s 
a 
According to Table 2, there was no significant difference in post-test scores of Group | (Game) and control group 


*<.05 
according to pre-test scores [F(1; 37)=1.702, p>.05]. 


Table 3. Group 2 (Fill in the blanks) and Group 6 (Control Group) 


Source of | Sum of Degree of Squares Partial Eta-Square 
Variance Squares | Freedom Mean 

219,439 109,720 

218,214 218,214 24,726 | ,000 


“ 
= 
— 


*<.05 


According to Table 3, there was no significant difference in the posttest scores corrected according to Group 2 (Fill in 
the blanks) and pre-test scores of the control group [F(1; 37)=.962, p>.05]. 
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Table 4. Group 3 (Ranking) and Group 6 (Control Group) 

Source of | Sum of Degree of | Squares F Partial Eta- 
Variance Squares | Freedom Mean Square 
322,162 161,081 18,944 
240,937 240,937 28,335 
_ = . 
= ~ a 
— Pf 


*<.05 
According to Table 4, there was no significant difference in the post-test scores of Group 3 (Ranking) and control 


group according to pre-test scores [F(1; 37)=2.118, p>.05]. 


Table 5. Group 4 (Pairing) and Group 6 (Control Group) 


Source of | Sum of Degree of Squares F Partial Eta-Square 
Variance Squares | Freedom Mean 

280,527 140,264 

262,302 262,302 


— 


10,520 
ao = 
*<.05 


According to Table 5, there was no significant difference in posttest scores of Group 4 (Pairing) and control group 
according to pre-test scores [F(1; 37)=3.203, p>.05]. 
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Tablo 6. Group 5 (True- False) and Group 6 (Control Group) 


Source of | Sum of Degree of Squares Partial Eta-Square 
Variance Squares Freedom Mean 
270,885 135,443 
269,285 269,285 
115 


et a - 


*<.05 


According to Table 6, there was no significant difference in the post-test scores of Group 5 (True-False) and the pre- 
test scores of the control group [F(1; 37)=.115, p>.05]. 


When these results were examined, it was seen that there was no significant difference between the experimental 
groups and the control group in terms of post-test scores corrected according to pre-test scores. 
The results of the analysis for the question of “How effective the learning activities of selected learner-content 


learners are at perceived learning levels?” are given in Table 10. 


Table 10. Comparison among groups in terms of perceived learning 


TEST GROUPS PERCIEVED LEARNING 
GROUP | (Game) t(37)= -1.081, p>.05 
GROUP 2 (Fill in the blanks) t(36)= .039, p>.05 


GROUP 3 (Ranking) t(36)= .273, p>.05 
GROUP 4 (Pairing) t(37)= -.182, p>.05 
GROUP 5 (True-False) t(37)= .014, p>.05 
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In addition, the same analyzes were repeated in terms of the student satisfaction variable. The results are given in 
Table 11. 


Table 11. Comparison among groups in terms of satisfaction 


TEST GROUPS SATISFACTION 
GROUP | (Game) t(37)= .443, p>.05 


GROUP 2 (Fill in the blanks) t(36)= .043, p>.05 


GROUP 3 (Ranking) t(36)= .527, p>.05 
GROUP 4 (Pairing) t(37)= -.257, p>.05 


GROUP 5 (True-False) t(37)= .862, p>.05 


When the answers of the participants to the questions about their satisfaction and perceived learning status were 
examined, it was seen that there was no significant difference between the 5 groups and the control group. 


Results 


In the light of the information obtained, it was observed that the learner-content interaction activities chosen in this 
study did not make a significant difference in the access of the learners to verbal information. Also, it was observed 
that these activities did not make any significant difference in the perceived learning levels and satisfaction of the 
learners. 


The lack of success difference between the interaction groups and the control group can be explained by the high level 
qualifications of the control group learning material. When the pre-test and post-test success scores were examined 
(between 7.90 and 10.75 on 17 full points), it was observed that all groups including control group had a similar 
increase. Therefore, these results are considered to shed light on the identification of learning environments where the 
types of interactions covered in the study are more effective and necessary. 


Another factor explaining that there is no difference is thought to be the content area used. The basic knowledge of 
English as the content of the learning material is selected. The scale of Turkey's target audience is also known to have 
low levels of English. Therefore, the fact that success change in English language skills, which is already hard to learn, 
has not increased enough after a short learning experience in this study, may be the reason why there is no difference 
among the groups. 


113 


Support Services in Open and Distance Education: 
An Integrated Model of Open Universities 


E. Genc Kumtepe 
Anadolu University, College of Open Education, Turkey 


E. Toprak 
Anadolu University, College of Open Education, Turkey 


A. Ozturk 
Anadolu University, College of Open Education, Turkey 


G. Tuna Buyukkose 
Anadolu University, College of Open Education, Turkey 


H. Kiline 
Anadolu University, College of Open Education, Turkey 


I. Aydin Menderis 
Anadolu University, College of Open Education, Turkey 


Abstract 


Support services are very significant elements for all educational institutions in general; however, for distance 
learners, these services are more essential than traditional (face-to-face) counterparts. One of the most important 
reasons for this is that learners and instructors do not share the same physical environment and that distance learning 
settings generally require intrapersonal interactions rather than interpersonal ones. Some learners in distance learning 
programs feel isolated because of this geographical and transactional distance. Furthermore, some fail to feel a sense 
of belonging to the institution because of lack of self-management skills, lack of motivation levels, and the need of 
being socialized, so that they are more likely to fail or drop out of an online class. In order to overcome all these 
problems, support services have emerged as a critical element for an effective and sustainable distance education 
system. 

Within the context of distance education support services, it is natural to include technology-based and web- 
based services and also the related materials. Moreover, institutions in education sector are expected to use information 
and communication technologies effectively in order to be successful in educational activities and programs. In terms 
of the sustainability of the system, an institution should provide distance education services through ICT enabled 
processes to support all stakeholders in the system, particularly distance learners. 

In this study, it is envisaged to develop a model based on the current support services literature in the field 
of open and distance learning and the applications of the distance higher education institutions. Specifically, content 
analysis technique is used to evaluate the existing literature in the distance education support services, the information 
published on websites, and applications of distance higher education institutions across the world. A total of 60 
institutions met the inclusion criteria which are language option (English) and availability of materials in the websites. 
The six field experts contributed in brainstorming process to develop and extract codes for the coding scheme. During 
the coding process, these preset and emergent codes are used to conduct analyses. Two coders independently reviewed 
and coded each assigned website to ensure that all coders are interpreting the data the same way and to establish inter- 
coder reliability. Once each web page is included in descriptive and relational analysis, a model of support services is 
developed by examining the generated codes and themes. It is believed that such a model would serve as a quality 
guide for future institutions, as well as the current ones. 


Keywords: Support services; open education, distance learning; support model. 
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LITERATURE REVIEW 


In the context of education, support services are defined as the whole of the services provided for the 
formation of learning communities and the continuation of the communities created. In the open and distance 
education, the current literature on support services has been observed to focus on the learner support services. One 
reason for this is that the pairs of learner-learner, learner-instructor and learner-institution are separated by space 
and/or by time. Distance education is a model of education that takes learners to the center of the education and tries 
to enable interaction between individuals, resources and institution. As stated at the beginning, it is understandable 
that support services are concentrated on learning and learner mainly when the distance from the system is considered 
physically and/or temporally. For example, Wright (1991) describes learner support as an inevitable element in 
creating an effective learning experience for distance education. Similarly, Thorpe (1988) defines support as an 
element of an open learning system that can respond to a particular individual's learning needs and process. The learner 
support in both distance education research and practice is influenced by the change in information and communication 
technologies, but only from the individual learners or autonomous learners, to the social learning and interpersonal 
learning environment have become a cover. In other words, the interaction and the support services of the individual 
that we witnessed in the earlier years of distance education through educational materials and environments have 
turned into a form that supports both individual-oriented (interaction with the individual and his/her own learning 
systems) and intrapersonal interactions (learner-learner, learner-instructor, learner-content-material/interface, etc.). 

In fact, support services are not only learner oriented, but all stakeholders of the system have an inclusive 
structure. These can be listed as faculty members or academic staff who are responsible for producing and delivering 
educational contents/materials; the administrative personnel in charge of administrative processes and technical 
personnel in charge of technical works in the system. In addition to studies focusing heavily on learning support in 
the literature, Padgett and Conceicao-Runlee (2000) pointed to the support of academic staff in the success of any 
distance education system. Similarly, Floyd and Powell (2004) reported that learning and instructional support create 
a positive interaction environment between learner and the whole staff. Researchers have stated that this positive 
integration can provide a successful model for effective teaching and learning. Therefore, all individuals and structural 
resources within the distance education system should be considered an integral part of the support services. In short, 
support services in distance education can be defined as the whole set of services provided to assist all stakeholders 
in the system (learners, instructors, administrative and technical staff, etc.) to use the resources of the institution 
effectively and thereby create quality learning. 


Main Components of Open and Distance Education Support Services 

Within the scope of the study, we also mentioned the basic elements that comprise the support services and 
the ways in which they are addressed in the research and practices. Support services in distance education have been 
categorized in different ways and what services should be concentrated in order to achieve this support. In fact, these 
classifications contain important concepts in terms of definition and inclusion of support services. The models of 
support services discussed in the literature are summarized in Table 1. Hui (1989) indicates learner support as 
individual learning materials, learning system, measurement and evaluation, practice. 


Table |. Support Services Models in Literature 
Year Researchers Components of Support Services 


1989 Hui e Individual learning materials 
e Learning system 
Measurement and evaluation 


1992 Rowntree e = Pre-program 
During program 
After program 


1995 Berge Pedagogical support 
Social support 
Administrative support 


Technical 


1997 Keast e Academic support 
e Administrative support 
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2002 McLoughlin 

2002 Simpson 

2003 Keegan 

2003 Tait 

2003 Rekkedal & Qvist-Eriksen 


Rowentree (1992) has divided the support services into three phases, pre-program and post-program, focusing on the 
instructional process. Berge (1995) has grouped roles that must be undertaken for the successful execution of open 
and distance learning as pedagogical, social, administrative and technical support. This classification of Berge is a 
common model that is used by many institutions. Another comprehensive support model in distance education was 
proposed by Keast (1997). According to Keast's model, learning support is discussed in five sections including 
administrative support, instructional support, technical support, consultancy and library support. 


RESEARCH METHOD 


This study is used content analysis technique to analyze open and distance (ODL) higher education 
institutions’ web pages in terms of their support services. Purposeful sampling process was applied to select 
institutions. The 60 institutions with the language option English or Turkish were determined. Research process is 


summarized in Fig. 1. 


Technical support 
Consultancy 
Library support 


Social support 
Peer support 
Task-related support 


Academic support services 
Non-Academic support services 


Information process 
Guidance process 
Registration process 
Integration process 
Final process 
Accreditation process 
The guidance process for post- 
program phases 
Academic support 
o Cognitive support 
"Teaching 
= Development of 
learning skills 

=» Assessment 

Non-academic support 
o Emotional support 
o Corporate support 

Pre-program support services 
Learning process support services 
Support services for post-graduation 
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Development of Preset codes 


Data analysis 


Fig. 1. Research Process of the Study 
FINDINGS 


Profiles of institutions 

This section summarizes the general profile information of the 60 institution that was examined. More than 
half of the institutions (n = 37; 60%) are located in the Asian continent, as the geographical distributions of the 
institutions will be seen from Figure 2. 


Africa Ocenia 


2% Ei 
America 12% saat 


3% 


Asia 
60% 


Fig. 2. Geographical distributions of open and distance higher education institutions 


Following Asia, 11 countries in Europe represented other institutions in the sample. The study included 
institutions in Africa (n = 7), America (n = 2), and Oceania (n = 1). Accordingly, it can be said that open and distant 
education is concentrated mainly in Asia, Europe and Africa. One reason for this is closely related to the history of 
open education. 

The distribution of higher education institutions in Asia which provides open education service is given in 
Figure 3. In Asia, China, India and Korea are among the most six institutions in the continent. These countries are 
followed by Philippines, Malaysia, Pakistan, Thailand and Vietnam with two institutions. 
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Number of Institutions 
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Fig. 3. Distribution of institutions in Asia by country 
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Figure 4 shows the distribution of open education institutions in Europe. There are also three open education 
colleges that provide higher education services in Turkey. These include Istanbul University, Atatiirk University, and 
Anadolu University, which has been providing open education services in Turkey for more than 35 years. In Germany, 
the University of Fern and the Bavarian Virtual University are the institutions that provide open education services. 
There is only one open higher education institution in each of the other countries shown in the Fig. 4. 
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Fig. 4. Distribution of institutions in Europe according to countries 


The number of countries and open education institutions located on other continents is shown in Table 2. 
Previously stated, among the 7 institutions, Africa is among the continents offering the most open higher education 


services. 


Table 2. Distribution of open higher education institutions by continents 


Continent 


Country 


Number of institutions 


Oceania 


America 


Africa 


Australia 

Canada 

USA 

Kenya 

Mauritius 

Nigeria 

Zimbabwe 

Sudan 

Tanzania 

Republic of South Africa 


my gee ga a Oe 


Total 
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In Figure 5, the founding years of the institutions were grouped with ten-year period. As the graph suggests, 
an increase in the number of institutions established over the years has been observed. The institutions established in 
1969 and earlier constitute 8.3% of all institutions. In this period, 5 institutions are located on different continents. 
1970-79 in this area, 9 institutions (15%) started to provide open education services. Again in 1980-89, a situation 
parallel to the previous 10-year period was observed and 10 institutions (16.7%) were founded in the period in 
question. In 1990 and later, 36 institutions (60%) in total are the educational institutions that serve in this area. 
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Fig. 5. Distribution of institutions’ years of establishment (n = 60) 


In the open and distance learning literature, institutions with more than 100,000 students are called as mega 
universities. Therefore, the student number of the institutions that provide open higher education services in this 
section are grouped in two parts (Fig. 6). Approximately 61% of the institutions show a change in the number of 
students under 100,000, while approximately 40% are in the group called mega university. Anadolu University is 
among these mega universities with the number of students over 1.5 million. 


100.000 = 
39.29%, 


Fig. 6. Distribution of numbers of students (n = 56) 


Finally, the structure of the programs given in these institutions has been examined and classified according 
to the teaching level (Fig. 7). It should be noted that there are at least one or more programs in each organization. 


Doctorate MAA 25 
Master: a 4s 
Bchlocr DL ss 
Associate CD 12 
Certificate LE 1s 


0 10 20 30 40 50 60 


Fig. 7. Distribution of programs in institutions according to the program level (n = 60) 


When this distribution is examined, undergraduate programs are given in the majority (97%), excluding 
two of the institutions. It is followed by Master's (80%) and PhD (42%) programs from postgraduate studies. 30% of 
the institutions have certificates and 12% of them have programs at the associate degree level. Massive open online 
courses (MOOCs), which have been popular in recent years, have begun to become widespread in some institutions 
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including Anadolu University. The following figure represents the integrated model of support services for open and 
distance education. 
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RESULTS AND SUGGESTIONS 


Support services are an important mechanism for the sustainability of both formal and distance education 
institutions, but learners in open and distance education need more support than the traditional education system. 
Some of the most important reasons for this are the fact that learners are physically separated from each other, 
from the system, from instructors, and that they provide interaction with all these actors and resources through 
open and distance communication technologies. The aim of the study in this direction is to examine the support 
services of the higher education institutions operating in the field of ODL in the world and the diversity of services 
they offer. In particular, this study has been designed to develop an inclusive support service model on the literature 
of existing support services in the field of ODL and applications of distance education institutions. 

The web-pages and current practices of the institutions were evaluated with content analysis technique. 
A total of 60 institutions have been included in the research. The coding scheme has been developed by six field 
specialists working in the field of open and distance education. When the descriptive and relational analysis for 
each web page is complete, a support services model is developed that includes the generated codes and themes. 
This model, presented in the findings section of the research, consists of two main sections, "learner" and "staff" 
support services. 

The learner support includes the support services given to the students who are candidates for being 
students and also the registered (current) students in the institution. The learner support services are one of the 
most important components of all institutions that are observed without exception on their websites. In the majority 
of institutions, there are guidelines for prospective students including information about introduction of the system, 
terms of registration and application. In some of the institutions, the support services documents mention the key 
elements and parameters of distance learning such as what the distance learner is, distance learning methods and 
strategies. Learning and consulting services for current learners are called academic support. In this section, there 
are contact information for academic staff in some institutions where students can contact the department or course 
basis questions. In most institutions, learning management systems (blackboard, Moodle, etc.) are integrated and 
teaching and learning materials are shared through these systems. Students are also provided with digital libraries 
as well as resource support. There is also support for the current learners, such as registration, scholarship status, 
student affairs, academic calendar, examination organization and other administrative issues. In addition, technical 
helpdesks and call centers that allow learners to access 7/24 in distance learning environments where information 
communication technologies are predominantly employed are the other support components observed in these 
institutions. Finally, both registered learners and alumni have established social support mechanisms. For 
registered students, online discussion forums in both corporate-supported learning environments as well as non- 
institutional social media (Facebook, Twitter, LinkedIn, etc.) are the environments that make social interaction 
possible. In addition, some institutions organize social activities such as student clubs, community service 
practices, student seminars, workshops, and sport events. For graduates, structures such as alumni union are 
important for both career development and alumni relationship with the institution The main objective here is to 
record the traces of the graduates in the business life and also to transfer their experiences as part of the system 
with different mechanisms (mentorship system, workshops, etc.) with existing students. 

The personnel component is a structure that covers the organization's "administrative" and "academic" 
personnel. 

Administrative and academic personnel are provided with "technical", and "administrative" support 
services, while academic support are also provided to the academic people in preparation, presentation, 
measurement and evaluation of educational materials. In addition, academic resources and research support are 
provided by the institutions in order to support academic research and professional contributions to the theory and 
practice in the field of ODL. 

The data obtained as a result of the research and the developed an integrated support services model are 
thought to be a guideline for designing and presenting support services to individuals and institutions that provide 
ODL services. This model has an important reference feature for higher education institutions that differ in the 
context of student size, academic discipline and diversity of programs. The number of students, regional and 
cultural differences cause each institution to develop its own unique business model and support services system. 
However, institutions may reorganize the indicators of these services presented here, taking into consideration 
human resources and structural resources in the organization. 

Support services should be considered as a separate unit/department in the organizational structure of the 
institutions that provide open and distance education. This unit, which is only to be positioned as support services, 
must be configured as an administrative unit that integrates with other relevant units in the organization. 

In accordance with the results of this study, the following topics should be considered for support services 
research. 


e Examining the variation of support services according to specific variables in the institutions. These 
variables; 
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e Learner profile (e.g., persons with disabilities, access to mobile devices, access to the Internet, 
etc.) 
Number of students 
Program level (undergraduate, graduate, certificate, etc.) 
Discipline (social sciences, natural science, etc.) 
e Cultural structure 

e Determining the satisfaction levels of learners about support services 

e Determination of satisfaction levels of administrative and academic staff for support services 

¢ Determination of measurable (quantifiable) indicators for each support services component 
In addition, it is assumed that the integrated model will be guiding the process of national and international quality 
and accreditation especially for our own organization. 
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Abstract 


Teachers have many options for continuing education and professional development. Online platforms are 
one option that can provide 24/7 access to professional development resources and communities for practice and 
support. By examining how teachers use these sites, we can offer suggestions for improvement and continued 
engagement. This paper describes an analysis of utilization patterns of instructional resources and site features in 
one online professional development platform for K-12 teachers in the state of Missouri in the United States. A 
clustering algorithm was applied to a large dataset of web analytics records to find patterns of use of professional 
development materials and site features. Prominent patterns suggest that teachers seek concrete video examples of 
teaching strategies and examples of units of instruction, and school administrators review teachers’ journals and 
perform administrative site functions. Teachers who are new visitors use the journal tool for initial reflection 
activities, but they do not access the journal tool in the future. Small patterns provide additional knowledge about 
the most utilized resources and the issue of referrer spam. 


Keywords: teacher professional development, educational data mining, web analytics, data clustering 
Introduction 


With increased access to the Internet, teachers have access to multiple opportunities for teacher 
professional development (PD) online. Carter (2004) identified four characteristics of a successful online teacher 
professional development: (1) 24/7 access to online resources, (2) community of learners, (3) community of 
practice, and (4) professional support. Each of these characteristics is designed to support the immediate PD needs 
of teachers through online self-paced resources. 

The EdHub Library is an online PD platform for K-12 teachers and school administrators maintained by the 
College of Education at the University of Missouri-Columbia. The EdHub Library is part of the Network of 
Educator Effectiveness (NEE). The Bill and Melinda Gates Foundation initially funded the EdHub Library in 2014 
in partnership with the University of Missouri-Columbia. NEE is a comprehensive educator evaluation system that 
tracks multiple measures of educator effectiveness, including classroom observation of teachers, units of instruction, 
professional development plans, and student surveys (Network for Educator Effectiveness, n.d.). 

Similar large online PD platforms exist, such as PE Central and MyTeachingPartner. PE Central is an 
online professional development platform for physical education teachers. With 162,000 visitors, PE Central 
provides lesson plans, best practices, classroom management, use of technology, and assessment ideas (Hanson, 
Pennington, Prusak, & Wilkinson, 2017). MyTeachingPartner (MTP) is another example of online self-paced 
resources where teachers have access to three types of resources that aim to improve teacher-student interactions: a 
video library of best practices of teacher-student interactions, a college course, and individualized web-mediated 
coaching sessions (MyTeachingPartner, 2017). 

The purpose of the EdHub Library is to support K-12 teachers, principals, university teaching assistants, 
teacher prep students in the advancement of best practices in teaching and learning in PK-12 and higher education 
levels (EdHub Library, n.d.). School districts with a subscription to NEE have access to (1) a video library of best 
practices in classroom teaching, (2) a video library of examples for scoring classroom observations, (3) a catalog of 
self-paced online modules, (4) copyrighted assessment instruments, and (5) a yearly principal calibration training. 
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In the EdHub Library, teachers can browse online resources from a list of topics and search content 
organized by teacher standards. EdHub can be accessed directly on the homepage at www. theedhub.org or through 
the NEE online teacher evaluation tool that tracks multiple measures of teacher effectiveness. Online resources are 
presented in modules with several activities that prompt teachers to reflect on their teaching practices or generate 
new ideas on instructional strategies for classroom implementation using the journal feature. Teachers have control 
over sharing journal entries with school administrators and reviewing their activity history from the user dashboard. 
In addition to the site features available to teachers, school administrators have administrative permissions that allow 
them to review teacher activity and control access within the school group. 

While we know about the purpose of the website and how many users are registered on EdHub, little is 
known about what users do once they are on the website. The purpose of this study is to discover the utilization 
patterns of content and site features using a clustering method and data visualization from 460,721 records available 
from October 2015 to February 2017. 

Google Analytics (GA) data are collected in the form of dimensions and metrics using a script embedded in 
the site and online resources. Dimensions are qualitative attributes of the data such as the page, type of browser, and 
location. Metrics are quantitative measures of data such as time spent and number of views of a page. To make sense 
of the data collected, combining dimensions and metrics generates insights from user interactions with online 
resources and platform features. 


Literature Review 


This section outlines existing literature related to the components of effective teacher professional 
development, evaluation of in-person and online PD, and research design for investigating utilization patterns of 
professional development platforms. 


What We Know About the Impact of Teacher Professional Development 


Multiple studies have investigated the components of teacher professional development and the impact of 
PD on teaching practices and teachers’ professional careers. Desimone (2011) argued that the core features of 
effective teacher PD lie in content focus, active learning, coherence, duration, and collective participation. Borko 
(2004) described teacher PD as a contextualized educational system that consists of a PD program, facilitators as the 
providers of PD, and teachers as recipients of PD. Guskey (2002) argued that the ultimate goal of staff development 
is to address the motivational aspect of teachers engaging in professional development. A change in teachers’ beliefs 
and attitudes occurs when teachers participate in PD and change their classroom practices when teachers see 
evidence of increased student learning outcomes. 

Bechtel and O’Sullivan (2006) described contextual aspects in schools that affect teacher change, including 
school culture, micropolitics of schools, level of support, and workplace conditions. Coldwell (2017) argued that 
highly engaged teachers in PD are very likely to stay in the profession because they are given opportunities to 
improve and validate their knowledge that makes them more motivated and confident as science teachers. Rice and 
Dawley (2009) surveyed 259 teachers, administrators, and instructional coaches from American K-12 virtual schools 
to understand professional development practices. Their findings suggest that online teacher professional 
development should be characterized by offering (1) multiple opportunities for professional development, (2) 
ongoing training sessions, (3) coach or peer mentoring sessions, (4) student-focused activities, (5) customized 
pieces of training based on teacher needs, and (6) trainings aligned to standards. 


Evaluating Teacher Professional Development 


While multiple studies investigated the characteristics of teacher professional development from the teacher 
perspective, the following literature describes evaluation studies of high quality in-person and online teacher 
professional development. 

Lindvall, Helenius, and Widerg (2018) argued that high-quality teacher professional development exhibits 
a level of coherence or alignment between a teacher’s current knowledge and the proposed teacher change from the 
PD that is suitable for his or her current knowledge and skills. Abell et al. (2007) constructed profiles of professional 
development programs that aimed at building contextual factors in designing and implementing effective teacher 
professional development. 

With the advent of educational technologies, teachers have multiple venues to engage in professional 
development opportunities online that allow them to address their immediate classroom challenges (O’Brian, 2016). 
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The following literature identifies research in online teacher professional development with an instruction that takes 
place 80% to 100% online based on the Online Report Card by Allen and Seaman (2016). 

Yoo (2016) investigated the effects of online professional development on teacher self-efficacy using the 
Teacher’s Self-Efficacy Scale (TSES) developed by Tschannen-Moran and Woolfolk Hoy (2001). Participants 
reported increased self-efficacy regarding (1) enhancement of teaching skills, (2) attainment of professional goals, 
and (3) acquisition of new pedagogical knowledge. 

Frey (2009) investigated the effects of project-based online professional development experiences on the 
performance of K-12 students with disabilities and teachers’ instructional practices. The qualitative analysis of 
discussion boards and journal entries suggested that (1) teachers improved their skills related to PD activities, (2) 
students with disabilities exhibited improved classroom performance, and (3) improved student performance 
affected teachers’ readiness to implement evidence-based instructional practices. 


Understanding Resource Utilization with Clustering of Digital Libraries, Online Courses, and MOOCs 


The researcher’s approach to analyzing large web analytics data is to use a clustering algorithm to derive 
utilization patterns of resources and site features. Several studies have used the simple k-means algorithm to 
understand patterns of educational digital libraries, online courses in higher education, and massive open online 
courses (MOOCs). Khoo et al. (2008) examined user behavior by analyzing the session length web metrics produced 
by various educational digital libraries. Bollen and Luce (2002) assessed the user characteristics and retrieval 
behaviors in a digital library by generating clusters to describe research interests and preferences from document 
retrieval networks. Valsamidis et al. (2012) utilized the simple k-means clustering method to generate patterns of 
online course quality. Rodrigues et al. (2016) performed a cluster analysis of student engagement patterns in a 
massive open online course (MOOC) and identified levels of student engagement regarding activity completion, 
number of comments, and answers in discussion forums. 

While existing research identifies positive effects of online teacher PD and success factors in high-quality 
online PD environments, only handful studies have documented user behavior in large online teacher professional 
development platforms. The need to analyze online behavior provides insights on how teachers use self-paced 
instructional modules and interactive site features in online teacher professional development environments. While 
web metrics are widely used in tracking user behavior and generate large amounts of data about resource utilization, 
the representation of web metrics data through a cluster analysis allows the observation of patterns from site features 
and resources when grouped based on similar characteristics. 


Statement of Purpose 


The purpose of this study is to discover the utilization patterns of the content and site features of a K-12 
online professional development platform. By data mining web metrics records from Google Analytics, this study 
allows a comprehensive examination of the patterns between new and returning visitors by applying the simple k- 
means algorithm and visualizing natural groupings or cluster outputs (Jain, 2010). 

The following research questions guided the design of the study: (1) What dimensions and metrics in 
Google Analytics are potential predictors of clustering? (2) What behaviors do new and returning visitors exhibit 
when using the platform? (3) What are the content and site feature patterns present from the predictor metrics and 
dimensions? The following section describes data extraction, manipulation, computation, and deployment of results. 


Methods 
Research Setting 
The research context is a large online content management platform with 50,425 K-12 teachers and school 
administrators seeking professional development in the state of Missouri in the United States. A total of 460,721 
web metrics records from the platform are available from October 2015 to February 2017. GA collects metric and 


dimension properties when a tracking script snippet is assigned to online resources and platform features. The web 
metrics reports of 17 months are available for extraction from the free edition of Google Analytics. 
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Data Mining Process 


This study adheres to the cross-industry standard process for data mining (CRISP-DM) methodology. The 
CRISP-DM is a popular data mining methodology that brings organization into data mining projects (Shearer, 
2000). This approach requires six systematic steps in any data mining project: (1) research understanding, (2) data 
understanding, (3) data preparation, (4) modeling, (5) evaluation, and (6) deployment. The following subsections 
describe each phase. 

Research understanding phase. The EdHub Library is built on Plone, an open-source content management 
system, which provides a mechanism for controlling user and content access. Each school year, access tokens are 
provided to teachers and principals to control access to the platform. At a system level, the EdHub Library is 
integrated with GA to track user and content activity. In this data mining project, the analysis aims at examining the 
patterns among users, instructional materials, and site features. 

Data understanding phase. To make sense of GA records, the site and content taxonomy was required to 
understand the overall structure of the EdHub Library. The taxonomy is derived from the web hosting indexing 
structure. Additionally, the researcher is responsible for creating and deploying instructional materials in the web 
repository in a structured manner. 

The Google Analytics Dimensions and Metrics Explorer provides descriptions of dimensions and metrics 
extracted from the platform shown in Table 1 (“Google Developers,” n.d.). 


Table 1. Google Analytics Dimension and Metric Descriptions 


A boolean, either new visitor or returning visitor, indicating if the users are new or returning 


a 
The total time (in seconds) users spent on a particular page 


Data preparation phase. After extracting 460,721 records in a tabular format, descriptive statistical 
measures were performed to identify missing cases. This preparation work makes it possible to create data 
visualizations in Tableau software to showcase (1) patterns between new and returning visitors, (2) the most 
searched terms, (3) the most accessed resources, and (4) the most accessed features. Tableau is a data visualization 
software that produces interactive data visualizations (Tableau, 2017). 
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Modeling phase. A feature selection was performed to understand the potential predictors of site utilization 
among the dimensions and metrics. The simple k-means algorithm was used to develop similar groupings or patterns 
for online resources and platform features among the predictor variables. The simple k-means algorithm was 
performed by running the dataset using 2, 10, 20, 30 and 40 cluster parameters by standardizing Euclidean distances 
with ten folds of cross-validation and 50 iterations. Euclidean distance is a measure of similarity and dissimilarity 
among data points. 

Evaluation phase. To validate the selection of the cluster parameter, the Davies-Bouldin validity index was 
used to evaluate the internal validity of clusters. The Davies-Bouldin validity index measures the average similarity 
between clusters. A lower validity index translates into a better cluster configuration regarding separation and 
compactness (Kovacs, Legany, & Babos, 2005). After manipulating several cluster parameters, a cluster parameter 
of 20 produced the lowest validity index. 

Deployment phase. The deployment phase involves presenting the data mining efforts to understand how 
teachers and school administrators use site features and resources. The data mining results are presented in data 
visualizations in the form of bar charts for searched terms, resources, and features. Also, visualizations are presented 
in heatmaps to show cluster classifications organized by user type and page. 


Results 


Seven out of twenty cluster patterns describe utilization patterns of resource pages and features for new and 
returning visitors. The remaining thirteen clusters overlap with the previous patterns. It is important to note that 
patterns do not represent a sequence of actions that occur in the site. Instead, patterns show the number of cases in 
which actions occur the most frequent and paint the ‘picture’ of how many events unfold in the site by examining 
prominent classifications or groupings. 

Pattern 1: new visitors - teacher or school administrator - user registration through the online teacher 
evaluation tool. This pattern shows new visitors, either teachers or school administrators, accessing the site through 
the online teacher evaluation tool to sign up for an account through the registration page on the EdHub homepage. 
This cluster shows classified cases with the following utilization pattern: (1) accessing EdHub through the online 
teacher evaluation tool, (2) signing up for an account using the login form, (3) reviewing content subscriptions, (4) 
verifying group access, (5) accessing content list of topics, (6) viewing group membership, (7) receiving a successful 
password reset, (8) reviewing teacher standards descriptions, (9) accessing terms and conditions on the registration 
page, (10) logging into the site, (11) getting an error with the wrong group access token, (12) reviewing content 
subscriptions, and (13) resetting a password. 

Pattern 2: returning visitors - teacher or school administrator - registered users through the online teacher 
evaluation tool. This pattern shows returning visitors, either teachers or school administrators, accessing the site 
through the online teacher evaluation tool. This cluster shows classified cases with the following utilization pattern: 
(1) accessing EdHub through the online teacher evaluation tool, (2) reviewing content subscriptions, (3) verifying 
group access, (4) signing up for an account using the login form, (5) viewing the group listing page, (6) logging out 
of the site, (7) reviewing teacher standards descriptions, (8) viewing token content access, (9) reviewing group 
subscriptions, (10) getting an error with the wrong access group token, (11) accessing online resources related to 
school administrators, (12) receiving a successful password reset, and (13) accessing online resources related to 
speech pathologist. 

Pattern 3: new visitors - school administrators - training, user subscription, and password reset. This 
pattern shows new visitors, school administrators, accessing a recertification training available to school 
administrators through the EdHub site directly. The recertification training allows school administrators to practice 
scoring teacher interactions in the classroom. To access the recertification training and copyrighted materials, users 
need specific content access the EdHub Library. This cluster shows classified cases with the following utilization 
pattern: (1) accessing recertification training, (2) accessing terms and conditions on the registration page, (3) 
resetting a password, (4) entering content token access, (5) reviewing teacher standards descriptions, (6) verifying 
group access, (7) reviewing content subscriptions, (8) accessing the library content list, (9) viewing members in a 
group, (10) entering the wrong password, (11) logging out of the site, (12) signing up for an account using the login 
form, and (13) viewing the group listing page. 

Pattern 4: new visitors - teachers - journaling and library content related to cognitive and affective 
engagement. This pattern shows new visitors, teachers, accessing journal entries and reviewing library content 
related to assessment topics, and teacher indicators related to cognitive engagement (1.2) and motivation strategies 
(5.1). This cluster shows classified cases with the following utilization pattern: (1) viewing journal entries, (2) 
accessing the library content list, (3) viewing the journal landing page, (4) reviewing teacher standards descriptions, 
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(5) reviewing content subscriptions, (6) reviewing NEE Teacher Standard 1| or content knowledge, (7) viewing the 
group listing page, (8) reviewing NEE Indicator 1.2 or cognitive engagement, (9) accessing assessment topics, (10) 
verifying group access, (11) reviewing NEE Indicator 5.1 or affective engagement, (12) signing up for an account 
using the login form, and (13) reviewing the about Edhub page. 

Pattern 5: returning visitors - school administrators - administrative viewing of journal entries. This 
pattern shows returning visitors, school administrators, accessing the site directly to view teachers’ journal entries 
and review library content related to instructional leadership topics and classroom observation videos. This cluster 
shows classified cases with the following utilization pattern: (1) viewing journal entries, (2) reviewing teacher 
standards descriptions, (3) reviewing user group preferences, (4) accessing assessment topics, (5) accessing online 
resources related to school administrators, (6) reviewing content subscriptions, (7) verifying group access, (8) 
viewing the group listing page, (9) accessing online resources related to instructional leadership topics, (10) 
reviewing teacher standard NEE Teacher Standard | or content knowledge, (11) accessing online resources related 
to administrator classroom observation training videos, (12) accessing formative assessment topics, and (13) 
accessing online resources related to speech pathologist. 

Pattern 6: returning visitors - school administrators - administrative group and user preferences. This 
pattern shows returning visitors, school administrators, accessing the group and user settings pages through the 
online teacher evaluation tool. These returning visitors have access to administrative settings to control group 
settings and member access. However, teachers do not have access to administrative settings. This cluster shows 
classified cases with the following utilization pattern: (1) viewing group members, (2) reviewing user preferences, 
(3) accessing group preferences through the online teacher evaluation tool, (4) logging out of the site, (5) accessing 
content knowledge topics, (6) accessing assessment topics, (7) accessing school administrators topics, (8) accessing 
professional development topics, (9) accessing formative assessment connection topics, (10) managing user 
subscriptions, (11) accessing administrator classroom observation training videos, (12) reviewing content 
subscriptions, and (13) reviewing NEE Indicator 1.2 or cognitive engagement. 

Pattern 7: returning visitors - teachers - group and user preferences. This pattern shows returning visitors, 
teachers, accessing the teacher group and individual user settings through the online teacher evaluation tool. In this 
particular pattern, site pages related to administrative settings are missing since teachers who are members of a 
group do not have access to group controls. Only school administrators have access to administrative settings. This 
cluster shows classified cases with the following utilization pattern: (1) reviewing user preferences, (2) accessing 
group preferences, (3) reviewing teacher standards descriptions, (4) reviewing content subscriptions, (5) viewing the 
group listing page, (6) accessing beginning teacher assistance topics, (7) accessing assessment topics for beginning 
teachers, (8) accessing assessment topics, (9) verifying group access, (10) accessing platform tutorials, (11) 
accessing assessment connection topics, (12) accessing modules in assessment connection topics, and (13) accessing 
professional development topics. 


Discussion 


In this study, 460,721 web analytics records from October 2015 to February 2017 were clustered using the 
k-means algorithm to develop content and site feature utilization patterns of an online K-12 teacher professional 
development platform. In this study, the free version of Google Analytics was used to extract thirteen variables 
(page, user type, browser, city, page depth, day of the week, new users, users, sessions, bounce rate, session 
duration, pageviews, and time on page). 

A feature selection was performed to discover the predictors of the page variable, which tracks the content 
and features of the platform. Based on the feature selection, city, browser, and users were not significant predictors 
of the page variable. Nine predictor variables for the page variable were found: (1) user type, (2) page depth, (3) day 
of the week, (4) new users, (5) sessions, (6) bounce rate, (7) session duration, (8) pageviews, and (9) time on page. 
By selecting the lowest Davies-Bouldin internal validity index, a cluster parameter of 20 was performed on the 
dataset to form utilization patterns by user type and page The site taxonomy of the platform was obtained from the 
web server to interpret of clusters. 


User Search Behavior 
While prominent clusters do not show specific search behavior of users, exploring what users typed in 


search queries allows further understanding of teachers’ needs for professional development. Users searched for 
examples of units of instruction, examples of professional development plans, and classroom training videos. 
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Users’ search queries highlight the need for teachers to find practical and concrete implementation 
examples of teaching strategies that model the skills in their classroom context. Research shows that teachers face 
challenges in implementing new teaching strategies rather than merely learning the theory behind teaching 
strategies. Ermeling (2010) argued that teachers look for instructional solutions that lead to detectable improvements 
in student outcomes. Joyce and Showers (2002) stated that teachers are likely to explore training that models a new 
skill that is close to their classroom setting. Studies by Peery (2002), Blank and De Las Alas (2009), and Darling- 
Hammond et al. (2009) explored the positive outcomes between teacher practice and student learning when teachers 
focus on training with similar teaching skills and practices. 


Deriving Use Cases from New and Returning Visitor Cluster Patterns 


Seven patterns describe use cases for teachers and school administrators when seeking PD and performing 
administrative functions. Patterns 1 and 2 are related to new and returning visitors, either teachers or school 
administrators, obtaining a registration to EdHub through the online teacher evaluation tool, reviewing their group 
and content subscriptions, and resetting passwords. 

Pattern 3 describes new visitors, school administrators, accessing the EdHub site directly for training, 
checking their group subscription access, and performing password resets. In this particular pattern, these new 
visitors are school administrators going through a recertification training that requires administrative access to 
EdHub to verify teacher members in a school group. 

Pattern 4 describes new visitors, teachers, accessing the site directly to perform journal tasks, reviewing 
descriptions of teacher NEE Indicators 1.2 (cognitive engagement) and 5.1 (affective engagement), and accessing 
topics related to assessment or NEE Standard 7. The absence of administrative tools in this pattern indicates that 
these new visitors are teachers. 

Pattern 5 involves returning visitors, school administrators, accessing the site directly to view teachers’ 
journal entries, reviewing group and personal preferences, and viewing topics related to assessment, instructional 
leadership, speech pathology, and classroom observation training videos. 

Patterns 4 and 5 explain how teachers and school administrators behave on the site by discovering what is 
available in the library through the NEE Standards descriptions page or the library topic listing before performing 
journaling activities. Users can review descriptions of standards and their related topics from the NEE Standards 
page. Alternatively, users can review descriptions of PD topics through the library homepage. Regardless of how 
users access topics, users are presented with the alignment of a topic and standard to aid in the selection of journal 
activities. 

Pattern 6 involves returning visitors, school administrators, reviewing their group membership and personal 
preferences. In this particular pattern, school administrator access EdHub via the online teacher evaluation tool to 
review administrator PD materials, which is only accessible to school administrators, and manage administrative 
settings of a group. Regarding professional development pieces, this pattern contains topics related to assessment 
and professional development. 

Pattern 7 describes returning visitors, teachers, reviewing user preferences, group membership, and content 
subscriptions. Similar to pattern 6, teachers access EdHub via the online teacher evaluation tool to review teacher 
standard descriptions, getting started guides of the platform, and topics related to assessment, professional 
development, and beginning teacher. In this pattern, however, administrative pages are not present in the cluster. 


Examining Small Cluster Patterns 


Although prominent classifications are useful in understanding which site features and content occurred 
most frequently for the majority of users, small cluster patterns provide insights about the most utilized instructional 
resources and the issue of referrer spam. 

Small classifications for sessions. In the cluster output for sessions, small classifications with a high 
number of sessions appear among returning visitors in topics related to instructional strategies, examples of units of 
instruction, and critical thinking, whereas new visitors access topics about the process of evaluating units of 
instruction. 

Small classifications for page depth. In the cluster output for page depth, small classifications with a high 
page depth appear among returning visitors who access group preferences, whereas new visitors access the journal 
tool. 

Small classifications for pageviews. In the cluster output for pageviews, small classifications with a high 
number of pageviews are related to new visitors accessing video examples of indicators 1.2 (cognitive engagement) 
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and 7.4 (effects of instruction). Returning visitors have a high number of pageviews when accessing their user 
preferences page, and video examples of teacher indicator 4.1 (problem-solving and critical thinking) and 1.2 
(cognitive engagement). 

Small classifications for session duration. In the cluster output for session duration, small classifications 
with a high session duration can be observed in topics related to academic vocabulary for new visitors, and 
administrative group settings and recertification training for returning visitors. 

Small classifications for time on page. In the cluster output for time on page, small classifications with a 
high time on page suggest that new visitors spend more time on resources related to speech-language pathology, 
recertification training, and formative assessment, whereas returning visitors spend more time on topics related to 
instructional strategies and examples of units of instruction. 

Small classifications for referrer spam. Referrer spam is a troublesome issue in Google Analytics tracking 
that makes repeated website requests using fake links to the site being spammed for advertising purposes even 
though these spam websites never visit the target site. In Google Analytics, website administrators can turn on the 
bot filtering option to block web traffic from referrer spam. Even when the bot filtering option is checked in Google 
Analytics for this study, less than 3 percent of classifications show referrer spam websites for bounce rate, new 
users and session variables. The clustering algorithm was able to detect patterns of three referrer spam websites 
(traffic2cash, snip.to, and adf.ly) that sent advertising links to Google Analytics. Spammers aim to create backlinks 
for users to access spam sites inadvertently. 

While deleting spam records from web analytics records is ideal, two challenges are present in dealing with 
referrer spam. First, identifying referrer spam among thousands of records is difficult because these spam websites 
generate new subdomains or links to disguise their domains as new visitors. Second, cleaning spam website records 
is time-consuming and inaccurate because not all referral sites generate new traffic for spam purposes. For example, 
search engines and other sources, such as school districts networks and other online PD platforms, may redirect their 
searches to the EdHub domain. 

Even if the bot filtering option is enabled in Google Analytics, it is not guaranteed to eliminate all spam 
website traffic. It is important to note that referrer spam websites were not able to access and alter web analytic 
records because the platform requires login credentials to access site features and content. Since the clustering 
algorithm was able to detect very small classifications of spam websites while the bot filtering option was checked, 
proactive measures can be taken by creating specific filters to prevent these spam website domains from sending 
backlinks in the future. 


Comparing Tasks from New and Returning Visitor Cluster Patterns 


By comparing the seven prominent cluster patterns, it allows further exploration of differences in tasks and 
site access performed between teachers and school administrators. As shown in Table 2, in four patterns out of 
seven, teachers and school administrators access the EdHub site through the online teacher evaluation tool. In the 
three remaining patterns, teachers and school administrators access the EdHub site directly. 

When accessing the EdHub site through the online teacher evaluation tool, new visitors who are teachers 
(1) register for an account and (2) verify group and content membership, whereas new visitors who are school 
administrators (1) verify group members, (2) check content membership, and (3) access training materials. 

When accessing the EdHub site directly, new visitors who are teachers (1) access the journal tool, (2) 
review teacher standards descriptions, and (3) search for content related to NEE Indicators 1.2 (cognitive 
engagement) and 5.1 (affective engagement), whereas new visitors who are school administrators (1) verify group 
members and (2) check content membership. 

A fundamental difference exists in the tasks for new teachers when accessing the EdHub site through a 
third party and directly. When accessing the site directly, new teachers tend to engage more with the site by 
accessing the journal feature, searching teacher standard descriptions, and seeking online PD based on teacher 
standards. When the site is accessed through the online teacher evaluation tool, new teachers perform the necessary 
tasks of registration, and checking their group and content memberships. For school administrators, either new or 
returning visitor, they perform administrative tasks regardless of access. 

Returning visitors who are school administrators accessing the EdHub site, either through the online 
teacher tool or directly, show more engagement with reading teachers’ journal entries, checking group preferences, 
and accessing various online PD topics. Returning visitors who are teachers, accessing the EdHub site through the 
online teacher tool, review assessment and beginning teacher topics. Unfortunately, returning teachers do not engage 
with the journal tool in the future. 
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Table 2. Comparison among cluster patterns by user type 


Pattern | User Type | User Role Point of Prominent User Tasks Most Utilized 
Access Instructional Resources 
1 New 1. 


Teacher or Teacher Register for an account 


School evaluation 
administrator | tool 


Returning | Teacher or Teacher 
School evaluation 
administrator | tool 

New 

New 


School EdHub 
administrator | homepage 


i ; 


Returning | School EdHub 
administrator | homepage 
Returning | School Teacher 
administrator | evaluation 
tool 


EdHub 
homepage 


through the online 


teacher evaluation tool 


Reset password 
Verify group and 
content membership 


Register for an account 


through the online 
teacher evaluation tool 
Reset password 
Verify group and 
content membership 


Access teacher standard 


descriptions 


Register for an account 
directly on the site 
Reset password 

Verify group and 
content membership 
Verify group members 


View journal tool 


View journal entries 
Access group 
preferences 


Verify group members 
Access group 
preferences 

Access personal 
preferences 
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Speech-language 
pathologist 


Recertification 
training 


Teacher standards 
descriptions 

Teacher indicator 
description for 
Indicator 1.2 (cognitive 
engagement) 

Teacher indicator 
description for 
Indicator 5.1 (affective 
engagement) 
Assessment topics 


Teacher standard 
description for 
Standard 1 (content 
knowledge) 
Classroom observation 
training videos 
Assessment topics 
Speech pathologist 


Administrator library 
Assessment topics 
Professional 
development topics 
Classroom 
observation videos 


Returning | Teacher Teacher . Verify group and . Teacher standards 
evaluation content membership descriptions 


tool . Access personal . Beginning teacher 
preferences topics 


Several online professional development topics are present across the patterns pertaining to (1) cognitive 
engagement, (2) formative assessment, (3) beginning teacher, (4) affective engagement, (5) administrator library, (6) 
speech-language pathologist, (7) getting started with EdHub, (8) classroom observation video examples, and (9) 
examples of units of instruction. However, the following topics are not present: (1) communication, (2) data 
analysis, (3) Common Core State Standards and Next Generation Science Standards, and (4) family and community 
involvement. All topics are available to teachers and administrators, except for the administrator library. 
Descriptions of these topics present in the patterns are shown in Table 3. 


Table 3. Topic Descriptions 


Cognitive engagement Strategies for engaging students with content and developing students’ deep 
thinking 


Formative assessment Key concepts of ongoing assessment and processes of setting clear student targets 


Beginning teacher Curated instructional modules covering assessment, classroom environment, 
cognitive/affective engagement, professional communication, and ethical behavior 


Affective engagement Motivation strategies that teachers use to influence students’ motivation 


Administrator library Training videos and copyrighted materials related to scoring teacher-student 
interactions in the classroom 


Speech-language pathologist | Roles and responsibilities, professional knowledge, and referral documentation and 
process. 


Getting started with EdHub Tutorials related to registration, subscriptions, and journaling 


Classroom observation video | A collection of exemplary videos showcasing teaching strategies by standard 
examples and grade level 


Examples of units of A collection of units of instruction examples shared by Missouri teachers 
instruction 


Communication Strategies for effective communication from teachers to students, parents, and 
administrators 


Data analysis Strategies for analyzing and building grade or building level data for instruction 


Common Core State Strategies for integrating Common Core State Standards and Next Generation 
Standards and Next Science Standards in the classroom 
Generation Science Standards 


Family and community Strategies for fostering adult involvement into the classroom 
involvement 
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Conclusion 


The results of this study can be used to understand user behaviors of teachers and school administrators, 
prioritize the development of professional development topics, and improve the navigation of the EdHub Library. 
Users will benefit from having quick access to the most utilized professional development materials when resources 
are placed on the homepage. As the Network of Educator Effectiveness grows, stakeholders will need to address the 
increased access to online professional development through the teacher evaluation tool and develop online 
professional development materials that target specific teaching skills with several classroom contexts. Teachers and 
school administrators perform the essential tasks based on their role. Teachers obtain a ‘birds-eye’ view of what is 
available in the EdHub Library before completing journal activities. Even though cluster patterns do not show 
returning teachers accessing the journal tool for future activities, cluster patterns show returning school 
administrators reading teachers’ journal entries and performing administrative functions. 
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Abstract 


This study examined the relationships between 146 sixth graders’ achievement and self-efficacy after they 
used a technology-enhanced problem-based learning (PBL) environment in a school with a high economically 
disadvantaged population. ANOVA measure indicated that this group of students’ science knowledge test scores 
increased significantly from pretest to posttest, while student self-efficacy decreased slightly. In addition, the results 
from multiple regressions showed that the higher the self-efficacy pre-scores, the higher the student science 
knowledge posttest scores. Furthermore, the qualitative analysis on student interview data showed that majority of 
the students had positive attitudes towards the PBL environment and gained more knowledge on the content 
compared to other skills after using the technology-enhanced PBL environments. 


Relevant Literature 


Emerging technologies have provided designers the opportunities to create technology-enhanced problem- 
based learning (PBL) environments. Studies have reported that these technology-based PBL environments can help 
increase student self-efficacy, critical thinking, communication, and collaboration skills (Brown, Lawless, & Boyer, 
2013; Lajoie et al., 2014; Neo, M. & Neo, 2001; Sendag& & Odabasi, 2009). For example, Neo, M. and Neo (2001) 
studied 46 college students who were asked to design a PBL using multimedia technologies in their class. By 
embedding student learning in this real-world problem, students were expected to “exercise analytical, critical and 
creative thinking in their work” (p. 21). The post-survey and interview data suggested that students became active 
participants and gained collaboration experiences in their learning process. Lajoie and her colleagues (2014) also 
examined cross-cultural medical students—in Canada and Hong Kong—during PBL using online digital tools and 
videos. In this study, they used AdobeConnect to facilitate synchronous communication among students, and the 
results indicated that the affordance of the technology provided opportunities for just-in-time communication 
without having students to travel across the globe. 
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For middle school students, literature has shown promising benefits after students used these technology- 
enhanced PBL environments—such as increasing student self-efficacy, learning interests, problem-solving, and 
collaboration skills (Brown et al., 2013; Sanchez & Olivares, 2011). In a recent review of literature, Merritt, Lee, 
Rillero, and Kinach (2017) summarized nine experimental studies to explore the effectiveness of PBL for K-8 
students (ages 3—14) in mathematics and science classrooms. Their review suggested that PBL was an effective 
instructional method for improving these students’ academic achievement, including knowledge retention, 
conceptual development, and attitudes. For example, one of the nine reviewed studies was conducted by Tandogan 
and Orhan (2007). They used both quantitative and qualitative methods to determine whether PBL would affect 
student academic achievement and concept learning in a 7" grade science classroom. The results indicated that PBL 
had positively affected student academic achievement and attitudes towards the science course. In addition, PBL 
positivity affected student conceptual development and kept student misconception at the lowest level. Karagalli and 
Korur (2014) also investigated academic achievement, attitude, and retention of knowledge of 9-11 years old 
students while they were using PBL. The results showed PBL had significant positive effects on student academic 
achievement, (F(1,112) = 46.78, p =.000) and knowledge retention (F(1,112) = 35.24, p = .000). 

Self-efficacy refers to the beliefs about one’s capabilities to produce designated levels of performance 
(Bandura, 1994). Research indicated student self-efficacy often acted as a predictor of academic achievement and 
motivation (Graham & Weiner, 1996; Jackson, 2002; Lane, J. & Lane, 2001; Pajares, 2003; Pintrich & DeGroot, 
1990). For example, Jackson (2002) found that for college students, self-efficacy was significantly related to their 
learning performance (i.e., exam scores in his study) on introductory psychology exams. Pajares (2003) specifically 
reviewed student self-efficacy regarding writing, and he pointed out that research findings have consistently shown 
that student writing self-efficacy are related to their writing performance. As for PBL, Dunlap (2005) suggested that 
college students had improved self-efficacy after completing a 16-week PBL course for software development. Yet, 
with respect to middle school students, researchers suggested that middle school students with lower self-efficacy 
tended to be reluctant to take on challenges presented in a typical PBL environment (Hsieh, Cho, Liu, & Schallert, 
2008). On the other hand, Brown et al. (2013) suggested technology-enhanced PBL environments had a positive 
impact on students’ science interest and self-efficacy. 

Despite many efforts, many urban, minority students lag behind academically compared to their white 
counterparts (Connell, Spencer, & Aber, 1994). Previous literature showed that PBL was used and enjoyed primarily 
by adult and adolescent learners who demonstrated comparatively higher levels of academic proficiency (Barrows, 
1987), and with gifted students (Stepien & Gallagher, 1993; Coleman, 1995). While low-income, minority 
population was not thoroughly studied regarding how they respond to the high academic challenge as presented by 
PBL. In one study, the researchers (Gordon, Rogers, Comfort, Gavula, & McGee, 2001) found that when used as an 
enrichment activity for just two percent of the curriculum in a public middle school that had 96 percent of low- 
income minority and low-performing students, PBL improved behavior and increased science performance of 
students, specifically on the active learning and teamwork. The individuals involved in PBL activities felt that the 
problem-based approach made learning more relevant and enjoyable. The effects of PBL were especially obvious 
middle school in science, but there were no consistent changes in academic performance in other subjects. However, 
this kind of PBL practices were not prevalent among economically disadvantaged students, partly due to insufficient 
teachers’ professional development. A study by Rillero et al. (2017) found most of the teacher preparation programs 
were inclined to focus narrowly on content knowledge teaching and specific methods with little focus on how to 
integrate PBL effectively. 

Given the benefits of using technology-enhanced PBL in improving student learning outcomes, and the 
lack of empirical evidence of its effectiveness in economically disadvantaged middle school students (Gallagher, S. 
A., & Gallagher, 2013), our study aimed to fill in the gap by investigating the use of technology-enhanced PBL for 
economically disadvantaged middle school students. 


Research Questions 


To understand whether technology-enhanced PBL environments can promote all students’ self-efficacy and 
science learning, we looked at achievement and self-efficacy of middle school students from a school with a 
minority and economically disadvantaged population. We asked these research questions: 

1. Are there any differences in students’ science knowledge and self-efficacy before and after they used a 

technology-enhanced PBL environment? 

2. Is there a relationship between students’ science knowledge and self-efficacy after they engaged in a 

technology-enhanced PBL environment? 
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Method 
Participants and PBL Environment 


Participants were 146 sixth graders from a middle school in a mid-sized southwestern city (Hispanic, 
53.1%; African-American, 13.4%; White, 26.1%; Other races,7.4%. See Table | for the demographic information). 
They used a technology-enhanced PBL environment designed for middle school space science as their curriculum 
over a three-week period in Spring 2017. Through inquiry-based activities, students learned about space science 
while practicing problem-solving, self-directed learning, and collaboration skills. See Figure 1 for the screenshots of 
the PBL environment. 


Table 1. 


Demographics of the Participating School 


Ethnicity % 
Hispanic 53.1 
African-American 13.4 
White 26.1 
Other races 74 


Figure 1. Screenshots of the PBL environment 
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Data Sources 


A 20-item science knowledge test, used in previous studies with similar samples engaged in the same PBL 
environment (Liu, Hsieh, Cho, & Schallert, 2006; Liu, Horton, Olmanson, & Toprac, 2011), was used to measure 
student understanding of the scientific concepts introduced in the curriculum (Cronbach’s alpha = .77). Eight items 
from the Motivated Strategies for Learning Questionnaire (Pintrich, Smith, Garcia, & McKeachie,1993) were used 
to measure self-efficacy (Cronbach’s alpha = .91). Both instruments were given before and after the use of the PBL. 

Qualitative data were also collected to understand the differences after students used this technology-based 
PBL environment, as well as the relationship between students’ science knowledge and self-efficacy. Students were 
asked to answer three questions in the post-survey, including: 

1) How would you describe the environment to a friend? 

2) What did you learn from the environment? 

3) Do you like this environment or not and Why? 

To ensure the trustworthiness of the qualitative data analysis, the survey responses were coded by one 
researcher first, and verified by other two researchers separately to reach the 100% agreement. A codebook was also 
generated during the analysis. 


Findings 


To answer research question one, ANOVA was used. The results indicated there was a significant main 
effect for the time of testing for science knowledge and self-efficacy: students’ science knowledge scores increased 
significantly from pretest to posttest (F(1,144) = 113.06, p < .0001, ES =.44; Mbretest = 42.67; Mposttest = 56.88); 
students’ self-efficacy decreased slightly: F(1,144) = 6.72, p < .01, ES =.05; Mpretest = 3.96; Mposttest = 3.80). 
Interestingly, girls had lower self-efficacy and their self-efficacy, though not significant, decreased slightly more 
than boys. However, overall, these changes did not depend on gender (see Table 2). 


Table 2. 


Students’ Science Knowledge Test and Self-Efficacy Scores 


Boys Girls Total 

Measure n M (SD) n M (SD) n M (SD) 
Science Knowledge Score 
(on 0-100 scale) ie ue a 

Pretest 45.34 (20.94) 40.00 (15.81) 42.67 (18.68) 

Posttest 58.63 (20.64) 55.14 (18.33) 56.88* (19.53) 
Self-Efficacy 73 73 146 
(Scale of 1-5) 

Pretest 3.96 (.79) 3.97 (0.59) 3.96 (.70) 

Posttest 3.84 (.78) 3.76 (0.82) 3.80** — (.80) 


* Significantly different from the pretest, P < .0001. 
** Significantly different from the pretest, P< .01. 


As for research question two, multiple regressions showed a significant moderate R of A8, F (2, 143) = 
66.66, p < .0001, with self-efficacy pretest scores significantly predicting science knowledge posttest scores while 
controlling science knowledge pre-test: b = 7.69, t(143) = 4.29, p < .0001. That is, the higher the self-efficacy pre- 
scores, the higher the students’ science knowledge posttest scores. 
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To further understand how this environment affected student self-efficacy and the relationship between 
student science knowledge and self-efficacy, the researchers analyzed the qualitative data. Based on the interview 
data, over half of the students (i.e., N= 77, 52%) would describe this environment positively to their friends. For 
example, one student suggested that she would tell her friends that “it is fun. interesting, and easy to use”. 
Admittedly, 10.8% of the students had negative comments about this environment, such as “It’s really boring. And 
that you get to use computers with it.” However, majority of the students showed positive attitude towards this PBL 
environment. See Table 3 for the definitions of codes and examples. 


Table 3. 


How would students describe the environment to a friend? 


Code Boy Girl Definition Code Example 
(N = 148) (N= 73) (N=75) 

Positive 40 37 Participant positive comments about it is fun. interesting. and 

(N=77) the environment, such as fun, good, and easy to use. 
interesting. 

Neutral 21 28 Participant comments that are neutral It’s ok 

(N = 49) about the environment. 

Negative 7 9 Participant negative comments about It’s really boring. And that 

(N = 16) the environment, such as Boring, you get to use computers 
Pointless with it. 

Null 5 1 Participant comments that are not valid go to google classroom 

(N = 6) for consideration. 


When asked what they have learned from the environment, students gave a variety of responses. These 
responses were coded into seven categories that emerged, such as solar system, game elements, scientific concepts, 
motivation, learning strategies, nothing and null (see Table 4 for the definitions of codes and examples). According 
to the data, students learned about the actual content knowledge the most (i.e., solar system, such as the planets, 
moon, space, and so on), followed by game elements and scientific concepts. For example, one student wrote “1 
learn about that there are different types of moons and i learn the planets,” which indicated he learned about the 
“Solar System.” Another student wrote “I also learned about the different possible species for aliens and where their 
home would be,” which was coded as learning about the “game elements.” A few students also indicated they 
improved their motivation and learning strategies, such as “What I learned in alien rescue [the name of the 
technology-enhanced PBL environment] was that I should pay attention more.” In addition, five students said that 
they did not learn anything in the environment, which accounted for 2.82% of the codes, such as “to be honest i 
didn’t really learn anything” 


Table 4. 


What have the students learned from the environment? 


Code Boy Girl Definition Code Example 
(N= 177) (N = 89) (N= 86) 
Solar System 45 60 Participants describe they 
(N= 105) learn about the solar system, i learn about that there are 
such as planets, moon, space, _ different types of moons and i 
and so on. learn the planets 
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Game 
Elements (N = 
27) 


Scientific 
Concepts 
(N= 14) 


Motivation 
(N= 9) 


Learning 
Strategies 
Wy) 


Null 
(N=7) 


Nothing 
(N=5) 


17 


10 


Participants describe they 
learn about the game elements 
in the environment, such as 
Aliens, Alien habitat, and so 
on. 


Participants describe they 
learn about the scientific 
concepts, such as instruments, 
probes, barometer, and so on. 


Participants describe their 
motivation elements, such as 
they had fun in the 
environment, or they got 
bored, and so on. 


Participants describe they 
learned the strategies, such as 
paying more attention, 
research, problem-solving, 
and so on. 


Participant comments do not 
make sense for what they 
have learned. 


Participants describe they 
learn nothing. 


I also learned about the different 
possible species for aliens and 
where their home would be. 


How to work with many different 
instruments like the seismograph, 
the infrared camera, and the 
barometer. I was able to research 
multiple things at once with 
another person. We worked 
together to get our research done. 


It was a fun way to learn new 
stuff 


What I learned in alien rescue 
was that I should pay attention 
more 


to be honest i didn’t really learn 
anything 


In the post-survey, we also asked the students whether they liked this environment or not. The codes 
indicated that the majority of the students (NV = 106, 72.6%) liked the environment. For example, a student wrote, 
“liked. It. A lot”. There were also 22.6% (N = 33) of them disliked the environment—“‘T hated it; It was boring.”(see 
Table 5 for the definitions of codes and examples). 


Table 5. 


Whether students like this environment or not? 


Code 
(N= 146) 


Like 
(N = 106) 


Dislike 
(N = 33) 


Neutral 
(N= 6) 


Boy 
(N= 73) 
55 


14 


3 


Girl 
(N= 73) 


51 


19 


Definition 


Participant comments on they like 


this environment. 


Participant comments on they dislike 


the environment. 


Participant comments were neural 


regarding the environment. 
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Code Example 


liked. it. A lot. 


I hated it; It was boring 


i like alien rescue and kinda 
not like alien rescue. 


Null 1 0 Participant comments that are not d 
(N= 1) valid for consideration. 


Discussions 


In support of the literature, the findings showed this group of minority and economically disadvantage 
students gained significantly more science knowledge after using the PBL environment. As for student self-efficacy, 
interestingly, girls had lower self-efficacy and their self-efficacy, though not significant, decreased slightly more 
than boys. However, these differences were not significant. In addition, the results also indicated that self-efficacy is 
an important factor affecting achievement for both boys and girls. Therefore, such findings suggest the importance 
of promoting student self-efficacy. 

With respect to qualitative data, over half of these minority and economically disadvantage students 
showed positive attitudes towards the technology-enhanced PBL environment. Most of them also commented that 
they liked this environment “a lot.” These positive attitudes could have contributed to their significant learning gains 
during the learning process. In addition, student explicitly expressed that they learned more about the solar system, 
which is consistent with the quantitative data showing that students had gained significantly more science 
knowledge. 

It is interesting that student self-efficacy slightly decreased despite they showed positive attitude towards 
the PBL environment. Therefore, there is a need to explore different approaches to promote student self-efficacy, 
such as improving the design of the technology-enhanced PBL environment or providing appropriate scaffolding for 
both teachers and students while they are using the environment. 

Further qualitative research could be directed to inquire female students’ overall learning experiences, 
especially their different feedback compared to male students. It will shed more lights on PBL environment design 
as well as scaffolding in terms of female students’ learning preference, thus to clarify the association between 
learning flow and their self-efficacy fluctuation. More research with longer time range and larger student population 
should be conducted, in order to reduce the novelty effects of such learning environments due to disadvantaged 
students’ limited exposure of PBL environments. 
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Abstract 


The purpose of this study is to investigate college students’ attitudes toward Recap® as part of their online 
language learning process. In creating the videos, students put into practice their Spanish language-learning skills. A 
14-question student attitude survey was used to rate the students’ perceptions of Recap® and how it related to their 
online language-learning process. The results revealed that students have positive attitudes toward the, students 
Recap® indicated that Recap® helped them to improve their language skills; and there was no significant gender 
difference in attitude toward the use of Recap®. 


Keywords: Spanish, Recap®, online class, higher education, oral presentation, language learning, technology. 
Background 


Technological tools offer many possibilities to second language teachers and to the teaching of the four 
language skills (Carrid-Pastor, 2016). These include listening, speaking, reading, and writing. Teaching a second 
language and measuring language ability in an online environment are often more challenging than in traditional 
face-to-face classes, especially in improving speaking skills since students and teacher are not physically together. 
Body language and facial expression are important aspects of communication, and these are lost when visual contact 
is compromised. 


Online courses in foreign languages must cover the same aspects of language learning as do face to face 
classes. Synchronous online communication tools, such as Skype, represent a convenient, low cost, increasingly 
common means of connecting two or more people across locations for social, professional, or educational purposes 
(Adinolfi & Astrid, 2017). However, they represent a few issues for students and teachers. One issue is that video 
chats are time-consuming for both the student and the instructor. Another is that students must coordinate 
availability among themselves for conversations. The video chat process increases the time commitment on the part 
of students for each chat activity. Students have an expectation of convenience and independence working when 
they take online courses, and video chats sometimes do not live up to this expectation. Finally, another concern of 
some educators regarding online courses is that students may not be doing the work themselves. Educators are 
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accustomed to seeing students face to face and thus become accustomed to recognizing the individuality of their 
student’s work. In the online setting, the educator may not truly know the work of their students. Recap® seeks to 
address these limitations of online courses in the areas of speaking and interacting with others in the language the 
students are learning. 


Apps are available in a variety of subjects which include: general early learning, math, astronomy, foreign 
language, literacy, study aids, art/creativity, geography, etc. (Shuler, Levine, & Ree, 2012). This dynamic app 
culture is growing fast in the education field and apps can be used to motivate student learning. Recap® is a 
question-led chat tool that uses student-created videos to facilitate oral production and interaction among students. 
Students can download Recap® to their smart phone to access class activities, record their own videos, listen to 
others’ recorded videos, and provide comments. Research has suggested that technology-infused approaches to 
pronunciation instruction may encourage the production of more culturally authentic language (Lomicka & Lord, 
2010). While using Recap®, the teacher can see the students’ faces and hear their voices by means of the videos. 
Although students cannot interact in a true conversation, they can however use the videos recorded on Recap® to 
comment on their own and other students’ videos. This enables the teacher to evaluate the students’ progress in 
mastering the four language skills. 


Research Questions 


1. What are student attitudes toward using Recap® in online elementary Spanish classes? 
2. What are student attitudes toward the effectiveness of Recap® in improving their Spanish language skills? 
3. Is there any difference between female and male students’ attitudes toward Recap®? 


Method 
Participants 


The participants were 69 undergraduate students who were enrolled in elementary Spanish 101 online 
courses from the Rocky Mountain region during the 2018 Spring semester. Most of the participants were English 
speakers who had little or no prior knowledge of Spanish. The group consisted of 23 males and 46 females. 


Materials 


An online student attitude survey that consists of 14-items was used with Likert-scale questions. The first 
10 questions asked students about the video-created tool itself, and the contribution of the video-created tool to their 
language learning. The remaining four open-ended questions asked about the students’ experiences and 
recommendations using Recap®. The survey was linked to an online survey platform known as Qualtrics®. 


Procedures 


The researchers have obtained approval from the Institutional Review Board to conduct this study. Through 
the video-created app, students created five videos on five different topics during the semester, approximately one 
video every three weeks. Students read the topic questions in Spanish and created a video of themselves responding 
in Spanish. Students then viewed their classmates’ videos and responded by writing comments in Spanish to the 
videos. During the last week of classes, the students completed an online attitude survey about the video-created tool 
and about their learning experience. 


Data Analysis 


To answer research question one and two, students’ responses to the first 10-item student attitude survey 
questions were analyzed using Exce/®. For positive stated items, strongly agreed is scored as 5, agree is scored as 4, 
undecided is scored as 3, disagree is scored as 2, and strongly disagree is scored as 1. Means and standard 
deviations for these items were calculated and reported. In addition, students’ comments in response to each open- 
ended question in the student attitude survey were categorized and reported. To answer research question three, a 
two-sample t-test was used to analyze the data. 
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Results 
Research Question 1: Student Attitudes Towards Using Recap® 


Table | shows the distribution of the 10 survey questions describing student’s attitudes toward Recap®. 
Data shows that the overall mean is 3.77; meaning that students had mostly positive attitudes toward using Recap® 
to learn the Spanish language. Additionally, the overall standard deviation for the survey is 1.21, which shows a 
small variation in the data collected. The data also shows that survey questions #2. Recap® was easy to use (M = 
4.30, SD = 0.91), survey question #4. Recap® helped me improve my speaking skills (M = 4.18, SD = 1.05), and 
survey question #1. Recap® met my expectations (M = 4.14, SD = 0.86) have the highest ranks across all questions. 
In contrast, survey questions #10. My classmates’ oral presentations helped me improve my writing skills (M = 
3.20, SD = 1.27), survey question #9. My classmates’ oral presentations helped me improve my reading skills (M = 
3.21, SD = 1.35), and survey question #5. Recap® helped me improve my reading skills. (M = 3.39, SD = 1.27) 
have the lowest ranks among all questions. 


Table | Rank of Survey Items 


1 2, Recap® was easy to use. 4.30 0.91 
2 4. Recap® helped me improve my speaking skills. 4.18 1.05 
3 1 Recap® met my expectations. 4.14 0.86 
4 7 My classmates’ oral presentations helped me to improve my listening 4.09 1.05 
skills. 
5 3 Recap® helped me improve my listening skills. 3.88 1.18 
5 8. | Myclassmates’ oral presentations helped me improve my speaking skills. 3.88 1.10 
7 6. | Recap® helped me improve my writing skills. 3.45 1.32 
8 5 Recap® helped me improve my reading skills. 3.39 1.27 
9 9 My classmates’ oral presentations helped me improve my reading skills. 3.21 1.35 
10 10. _My classmates’ oral presentations helped me improve my writing skills. 3.20 1.27 
Overall 3.77 1.21 


Generally, students have a positive attitude toward using Recap® as well as a positive opinion about the 
effectiveness of Recap®. A majority of the students participating in the survey (84%) reported that Recap® met 
their expectations and Recap® was easy to use (See Figure 1). 


1. Recap® met my expectations 2. Recap® was easy to use 
Disagree Disagree 
Undecided es Undecided 1% 


9% 9% 


Figure 1. Student attitudes toward using Recap® 
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In the final section of the survey, four open-ended questions were presented to the students, three of which 
were intended to discover the attitudes of students toward using Recap®. These three questions were worded to 
identify what the students thought were the advantages of Recap® (“What did you like most about Recap®?”), the 
disadvantages (“What did you like least about Recap®?’’), and any changes that they would make if they were an 
instructor using Recap® (“If you were the instructor, what suggestions would you make about using Recap®?”). 
The results of these questions revealed four common themes amongst the participants. The themes are presented in 
the order in which they were identified through the open-ended questions. 


Advantages of Recap® 


Each of the participants was asked to describe what they liked the most when using Recap®. Among the 
main features that students liked most were the easy interactive features provided by the app, such as connecting 
with the instructor and other students through comments and feedback and oral presentations using video and audio 
recordings. Phrases used in student comments included “providing comments, feedback, opinions”, “providing 
comments for reflection”, and “it was easy to comment”. Another feature is that they described Recap® as a user- 
friendly Recap® that can be easily navigated on computers and cell phones. Students commented “easy 
accessibility”, “fast-loading” and “free to download”. Some students said that Recap® was fun to use because of its 
color scheme and layout. Examples of students’ comments included “engaging color scheme” and “interesting 


layout”. 
Disadvantages of Recap® 


When asked about weaknesses of Recap®, many students reported technical troubles with Recap®, such as 
difficulties downloading Recap®, difficulties navigating Recap® on cell phones or computers, difficulties recording 
audios and videos. Also, many students reported that Recap® would “freeze” or “glitch” sometimes, and because of 
Recap® inability to save previous work, many students had to redo their assignments or recordings. Some students 
commented “freezes/shuts down/glitches” and “inability to save work after logging out”. Another main complaint 
about Recap® was Recap® lack of notifications. Some students found Recap® inability to provide reminders of 
upcoming assignments and projected as a problem. Additionally, the presentation of videos without support in 
English language and the requirement of entering an access PIN every time when the student needed to log-in into 
Recap® were viewed as significant difficulties. Examples of students’ comments included “it required PIN every 
time used with no ‘remember me’ option” and “lack of English transcripts or subtitles”. 


Changes on the Recap App® 


More than half of the students participating in the survey said that they won’t use Recap® differently. 
However, some students suggested changing assignment question structure and to decrease the number of repetitive 
questions in Recap® assignments. In addition, since Recap® has some technical problems, many students suggested 
providing a “backup” method to submit assignments and projects, such as using Canvas (a learning management 
system). Some of the main features students wanted to add to Recap® are “an assignment and project reminder” 
feature as well as a “conversation” feature where students have the opportunity to “chat” with each other to practice 
their speaking and listening skills. Other students stated that they would use Recap® differently by providing more 
specific instructions regarding assignment and oral presentations. Interestingly, on the one hand, some students 
wanted to eliminate the commenting on other students’ assignment and projects requirement and found it to be an 
impractical requirement. On the other hand, some students wanted to use the “comment” feature more and asked to 
enforce a minimum number of sentences per comment. 


Research Question 2: Recap®’s Impact on Students’ Spanish Language Skills 


For research question 2 “What are student attitudes toward the effectiveness of Recap® in improving their 
Spanish language skills?”, as illustrated in Figure 2, we can see that over 50% of the participating students either 
“strongly agree” or “agree” that Recap® improved their Spanish language skills in all four areas, including 
speaking, listening, reading, and writing. Also, 82% of students reported that watching or listening to their 
classmates’ oral presentation improved their listening skills while 68% reported that watching or listening to their 
classmates’ oral presentation improved their speaking skills. However, only 43% of students reported that their 
classmates’ oral presentations improved their reading and writing skills. 
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In addition to the Likert-type survey questions, each of the participants was also asked to describe how 
Recap® impacted their Spanish language skills (“How did using Recap® affect your language skills?”). Almost all 
students reported an improvement in their Spanish language skills. Due to the ability to practice speaking and 
listening to other classmates using Recap®, students were able to improve their speaking skills, which enhanced 
their Spanish vocabularies and increased their confidence in speaking Spanish as a second language. Students stated 
“ability to listen to other classmates speaking and learn from them” and “increases confidence in speaking”. Also, 
since Recap® provided the option of listening to other classmates with various accents and one-on-one feedback, 
many students reported improvement in their listening skills. Some students wrote: “Ability to listen to other 
classmates speaking and learn from them”. Improvement in other language skills, such as pronunciation, writing and 
sentence construction, and grammar and Spanish conjugation was also noticed by students using Recap®. 


3. Recap® improves listening skills 4. Recap® improves speaking skills 5. Recap® improves reading skills 


5 . Strongly Disagree — 
Strongly Disagree _ Disagree + Ys Strongly Disagree a \ 
5% \ 2% \ 3% 


& 


3 
eS 


6. Recap® improves writing skills 7. Oral presentations improve listening skills 8. Oral presentations improve speaking 


skills 


Strongly Disagree Disagree prey Disagree 


Disagree 
: 4% 4% 5% 


7% 
Undecided 


7% ~\ 


P 


9. Oral presentations improve reading skills 10. Oral presentations improve writing skills 


Strongly Disagree 


S 
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For research question 3 “Is there any difference between female and male students’ attitudes toward 
Recap®?”, when comparing male and female students’ attitudes toward Recap® in the class, there is no significant 
difference in attitude toward Recap®, as shown in Table 2 and Table 3 below. Therefore, it can be concluded that 
male students (M = 4.23) and female students (M = 3.67) have similar attitudes toward using Recap® as well as 
their opinions about its effectiveness on language learning. 


Table 2 Descriptive Statistics Comparing Female and Male Students 


Group Mean Variance Minimum Maximum 
Male 4.23 0.65 2.80 5.00 
Female 3.67 0.72 1.5 5.00 


Table 3 Two Sample T-Test 


Variable DF t-value p-value 95% Confidence Interval 
Gender 13.712 1.970 0.069 -0.050 1.167 


Educational Implications 


In this study we gathered, organized, and analyzed data which demonstrate that apps can improve the 
process of language learning. It was also demonstrated that students had a positive opinion of Recap® use and 
effectiveness in their own learning. Technology in the classroom opens up additional opportunities for learning than 
are traditionally recognized. Although our study only dealt with language learning, it is evident that apps can be 
effective and enjoyable in other areas of learning. 


This study may offer several implications for second language teaching. In addition to providing an 
overview of how to effectively cover the four aspects of language teaching, the results of this study could aid 
language instructors in a variety of classroom settings. Those settings may include face to face, hybrid, and online 
classes. 

Recap® need not be limited to language courses, since it is adaptable to different areas of study. Any online 
or hybrid course could benefit from the audio and video features and the opportunities for students’ oral 
presentations. 


Our study results were in line with other research results indicating positivity between using apps as an aid 
in various types of classes and language learning. According to Demouy and Kukulska-Hulme (2010) and Berns et 
al. (2015), technology apps in general aid students in the language learning process, and the use of mobile apps in 
language learning helps learners practice listening and speaking and is effective in improving these skills. In 
addition, a mixed study showed that integrating technology in language instruction resulted in a noticeable 
improvement in the students’ oral communication skills and that the students had a positive attitude towards the 
integration (Kirkg6z, 2011). Furthermore, a case study conducted at a Japanese university showed that students 
reported that recording themselves improved their language verbal skills by increasing their word count and 
becoming more fluent and confident when speaking English (Gromik, 2012). All the foregoing data support the 
results of previous studies which demonstrate how technology can be used to improve language learning. 


According to Davis (2015), providing opportunities for the asynchronous and synchronous communication 
between learners on the course is an important part for creating a sense of presence and building a learning 
community in an online course. Our study showed that Recap® helped students to create a ‘class presence’ among 
themselves, as they watched each other’s videos and participated in making and reading each other’s comments 
about the videos. Regarding gender, our study results indicated that there was no observable difference in attitudes 
toward Recap® between males and females. 
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In this study, only Recap® was considered, but there are many educational apps available. For future 
research, comparison of two or more of similar apps on language learning would be useful. Especially for the 
language teachers to integrate apps into their face to face classrooms or online courses. Additionally, our sample was 
somewhat limited in size; a study with a larger sample size, a longer-term study, or a study in which data are 
collected from various studies over time, would yield stronger data to determine effectiveness. 


References 


Adinolfi, L., & Astrid, L. (2017). An exploratory study of translanguaging practices in an online beginner-level 
foreign language classroom. Language Learning in Higher Education, 7(1), 185-204. 

Allan, B. (2016). Emerging strategies for supporting student learning. London: Facet. 

Berns, Anke; Palomo-Duarte, Manuel; Dodero, Juan Manuel; Ruiz-Ladron, Juan Miguel; Calder6n Marquez, 
Andrea. (2015). Mobile apps to support and assess foreign language learning. In Helm, Francesca; Bradley, 
Linda; Guarda, Marta; Thouésny, Sylvie (Eds), Critical CALL — Proceedings of the 2015 EUROCALL 
Conference, Padova, Italy (pp. 51-56). 

Carrid-Pastor, M. L. (2016). Technology implementation in second language teaching and translation studies: New 
tools, new approaches. Singapore: Springer. doi: 10.1007/978-98 1-10- 0572-5 

Davis, T. (2015). Visual design for online learning. San Francisco, CA: Jossey-Bass, a Wiley Brand. 

Demouy, V., & Kukulska-Hulme, A. (2010). On the spot: Using mobile devices for listening and speaking practice 
on a French language programme. Open Learning, 25(3), 217-232. 

Fuller, R. G., Frey, B. A., & Kuhne, G. W. (2011). Distinctive distance education design: Models for differentiated 
instruction. Hershey, PA: IGI Global (701 E. Chocolate Avenue, Hershey, Pennsylvania, 17033, USA). 

Gromik, N. A. (2012). Cell phone video recording feature as a language learning tool: A case study. Computers & 
Education, 58(1), 223-230. 

Horwitz, E. K. (2013). Becoming a language teacher: A practical guide to second language learning and teaching. 
Boston: Pearson Education. 

Keengwe, J. (2015). Handbook of research on educational technology integration and active learning. Hershey: 
Information Science Reference, IGI Global. 

Keengwe, J. (2018). Handbook of Research on mobile technology, constructivism, and meaningful learning. 
Hershey, PA: Information Science Reference. 

Lomicka, L., & Lord, G. (2010). Podcasting-past, present and future: Applications of academic podcasting in and 
out of the language classroom. Academic podcasting and mobile-assisted language learning: Applications 
and outcomes (pp1-20). Hershey, PA: IGI Global. 

Nilson, L. B., & Goodson, L. A. (2018). Online teaching at its best: Merging instructional design with teaching and 
learning research. San Francisco, CA: Jossey-Bass, a Wiley brand. 

Seiter, J. S., Peeples, J. A., & Sanders, M. L. (2018). Communication in the classroom: A collection of G.LF.T.S. 
Boston: Bedford/St. Martins. 

Starkey, L. (2012). Teaching and learning in the digital age. New York: Routledge. 

Thirunarayanan, M. O., & Pérez-Prado, A. (2005). Integrating technology in higher education. Lanham: University 
Press of America. 

Tomei, L. A. (2009). Information communication technologies for enhanced education and learning: Advanced 
applications and developments. Hershey, PA: Information Science Pub. 

Kirkgoz, Y. (2011). A Blended Learning Study on Implementing Video Recorded Speaking Tasks in Task-Based 
Classroom Instruction. Turkish Online Journal of Educational Technology, 10(4), 1-13. 

Wright, R. D. (2015). Student-teacher interaction in online learning environments. Hershey, PA: Information 
Science Reference. 


150 


Ensuring the Educational Success of Students With Special Needs Through 
Appropriate Assistive Technology Implementation 


Soonhwa Seok 
Korea University 
145, Anam-ro, Seongbuk-gu, Seoul 136-701 
Republic of Korea 
sunaseok@yahoo.com 


Boaventura DaCosta 
Solers Research Group 
1445 Dolgner Place 
Sanford, FL 32771 
bdacosta@solersresearchgroup.com 


Keywords: Assessment; Assistive Technology; Evaluation; Evidence-Based Practice; Instrument Development; 
Multidimensional Scaling; Special Education Professors; Special Education; Students With Disabilities 


Abstract 


The need for valid measurement of assistive technology (AT) outcomes as a quality indicator in special 
education has been called out in the literature. To date, outcome assessments associated with quality of AT have 
fallen short of satisfactory validation processes. Furthermore, existing research has examined AT from the 
perspective of the user, neglecting other important points of view, such as educators and preservice teachers at the 
secondary level. This study identified dimensions underlying AT practice for outcome assessment along with the 
clusters of validated items on each dimension from the perspective of professors preparing preservice teachers for 
careers in special education. Using multidimensional scaling, four participants rated 19,404 pairwise comparisons, 
with the results revealing four underlying dimensions and 11 clusters across the dimensions. While more research is 
warranted, these dimensions and clusters may be used as a foundation for the development of an assessment 
instrument to enhance the quality of AT outcomes for students with disabilities. 


Introduction 


There is a need for valid measurement of assistive technology (AT) device and service outcomes as a 
quality indicator (Graham & Warnie, 2012). While this need has been well documented in the literature for many 
years (e.g., DeRuyter, 1995; Smith, 1996), in recent years growing attention has been paid to validation and outcome 
measurement of AT in response to the demand for evidence-based practices in special education (Agree & 
Freedman, 2011; Bernd, Van Der Pijl, & De Witte, 2009; Seok & DaCosta, 2014). Even so, to date assessments 
related to quality of AT have lacked satisfactory validation processes or frameworks to measure AT outcomes 
(Lenker & Paquet, 2003; Lenker, Scherer, Fuhrer, Jutai, & DeRuyter, 2005). 


Research on outcomes assessment and validation of AT measures has mainly examined the quality of AT 
from the user’s perspective instead of other points of view, to include those of the AT subject-matter expert, 
experienced educator, and preservice teacher at the secondary level. This is not to suggest that the research to date 
has not made significant contributions. For example, an extensive body of research has established guidelines and 
standards for determining the quality of AT, such as the Functional Evaluation for Assistive Technology (FEAT; 
Raskind & Bryant, 2002). Yet, despite the importance of assessing the quality of AT devices and services (Lenker et 
al., 2010), there continues to be a shortage of literature focused on AT evaluation and subsequent validation from the 
perspective of the AT expert (Seok & DaCosta, 2014). 


151 


Toward this goal, this study identified the dimensions underlying AT practice for outcome assessment 
along with the clusters of validated items on each dimension from the perspective of the AT expert; namely, 
professors preparing preservice teachers for careers in special education, who not only have considerable knowledge 
about different facets of special education and the implementation and use of AT, but also play a substantial role in 
shaping future special educators. 


Method 
Participants 


The study was conducted at postsecondary institutions in the Midwestern United States. Participation was 
restricted to professors in special education, who had taught students with disabilities, with a minimum of three 
years’ experience in selecting and implementing AT for students with mild or moderate disabilities, as evidenced in 
six or more semesters of syllabi reflecting AT implementation practices in their course(s) and authorship of five or 
more articles, chapters, and/or conference proceedings on AT use. Based on these criteria, 10 assistant or full 
professors were identified using convenience sampling; four responded to the invitation to participate. While all four 
were regarded as experts in their respective areas of special education, two were considered authorities on learning 
disability, one on early childhood, and one on behavior disorders. 


Materials 


The study was based on Seok and DaCosta (2014), who identified valid items and factors for students with 
mild and moderate disabilities derived from the FEAT, comprehensive literature reviews, and other pertinent content 
(see Raskind & Bryant, 2002 for a detailed explanation of FEAT). Seok and DaCosta formalized 99 items using a 
strict, multiphase approach that were rated by 1,472 special education teachers, and then validated using factor 
analysis (see Seok and DaCosta, 2014, for a detailed explanation). These 99 items were used in the current study. 


Given the large number of pairwise comparisons, a website was developed to allow the participants to 
perform the evaluations at their own pace. The site presented the item to be evaluated while randomly looping 
through each of the remaining items, affording a way to rate each pairing. A random order was adopted to control 
for “a systematic item order effect” (Davidson & Sireci, 2000, p. 79). The item to be evaluated was shown at the top 
of the screen with the items to be compared appearing randomly one at a time below. A new item was randomly 
generated once the current item was rated against all the others. Comparisons were scored using a 9-point scale, with 
1 representing the most extreme degree of dissimilarity and 9 the most extreme degree of similarity. For ease of 
rating, each item was depicted in a different color than the previous one, offering a visual that the item being rated 
(against all the other items) had changed. 


Procedure 


Potential participants received invitation letters that included the URL of the website. Upon accessing the 
site, they were then presented with the purpose of the study and instructions for how to perform the pairwise 
comparisons. The potential participants were then asked for their agreement to participate in the investigation. 
Consent resulted in further instructions, comprising a pairwise comparison example, and prompting participants for 
their email address and a password, allowing for later login. (The email and password were only used to allow the 
participant to complete the study in multiple sittings, not for identification.) That is, with the URL, email, and 
password, participants could log in at any time, and continue their pairwise comparisons. Each login presented 
participants with the instructions and pairwise comparison example, and then took them to the next pairwise 
comparison to be rated. Permission to perform the study was granted by the ethics committee of a research center. 


Results and Discussion 


To determine if the participants recognized dimensions of quality of AT outcomes and if the configuration 
of the items was represented mathematically, each participant rated 4,851 comparisons, resulting in a dataset of 
19,404 pairwise comparisons (4,851*4). The subsequent exploratory data analysis focused on scale construction and 
dimensionality reduction using multidimensional scaling (MDS). The Statistical Package of Social Science (SPSS) 
20.0 was used to develop the MDS matrices. 
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Multidimensional scaling was chosen because the technique is accepted in educational research and has 
been used in studies examining learning (e.g., Davison & Sireci, 2000; Seok, 2009). In addition, studies of MDS 
have mathematically shown that sample size does not have a valid and significant impact on the results (MacCallum, 
1979; Rodgers, 1991; Stalans, 1995); finally, a single rater can carry out MDS pairwise comparisons (Stalans, 1995), 
making MDS well suited for the current study. 


The MDS of the proximity dataset was analyzed using PROXSCAL algorithms. Used to determine 
configurations of items in a low-dimensional space, the PROXSCAL algorithms minimized the normalized raw 
stress. Stress and fit measures revealed that the normalized raw stress was .07136, S-stress was .19190, dispersion 
was .92864, and Tucker’s coefficient of congruence was .96366, showing that the four-dimensional model fit the 
data well. 


That is, the findings revealed four dimensions resulting from exploring the content validity items. Table 1 
shows the final multidimensional configurations from the MDS analysis, revealing that the four dimensions 
extracted from the data are closely related to content validity (O’Neil, Sireci, & Huff, 2004) and contribute to 
framing the content of assessment (Lenker et al., 2005). 


Table | 
Final Positioning of Items in the Multidimensional Space 
Items Dimensions Items Dimensions 

1 2 3 4 1 2 3 4 
1 -.272 320 311 -.434 51 .069 558 352 -.126 
2 139 -.662 -.002 -.002 52 -.072 389 530 150 
3 371 -.547 -.149 .002 53 .476 .400 151 -.222 
4 -.036 -.572 136 .333 54 357 549 -.146 .090 
5 043 -.015 -.051 677 55 261 -.192 -.103 589 
6 -.214 115 013 -.636 56 -.096 -.035 .672 -.027 
7 -.099 -.665 .084 .004 57 532 326 -.212 -.167 
8 -.366 -.495 122 256 58 395 -.219 .010 -.508 
9 -.460 358 -.328 .096 59 465 -.123 413 -.237 
10 243 -.558 163 227 60 -.043 -.304 285 537 
11 -.427 475 042 .220 61 564 172 -.293 -.110 
12 -.189 563 -.050 324 62 -.154 -.277 562 213 
13 -.617 -.262 -.076 -.048 63 -.163 -.540 -.204 -.321 
14 -.651 -.025 -.104 147 64 -.236 037 035 635 
15 -.229 453 -.377 229 65 384 -.216 354 377 
16 -.494 -.087 -.452 051 66 -.341 142 -.204 -.536 
17 -.185 456 -.380 -.276 67 -.001 392 165 -.533 
18 -.027 -.367 -.029 -.569 68 O11 -.487 -.464 -.076 
19 128 -.246 581 214 69 300 364 -.345 344 
20 -.275 564 .240 -.078 70 -.066 -.107 565 -.353 
21 -.090 239 554 -.301 71 -.087 301 -.592 -.083 
22 -.085 621 -.046 -.259 72 219 -.253 -.553 175 
23 -.335 -.172 -.341 -.454 73 -.204 232 -.148 592 
24 .063 380 -.542 132 74 .664 .004 -.072 -.114 
25 -.532 .270 -.250 -.196 75 372 -.504 .233 -.109 
26 -.531 -.253 144 -.305 76 210 244 .184 570 
27 -.406 -.185 506 -.064 77 365 167 -.542 054 
28 098 332 -.297 -.509 78 -.041 -.362 -.359 -.450 
29 419 114 429 -.296 79 -.621 078 -.232 -.103 
30 361 274 219 -.456 80 165 085 -.633 -.148 
31 603 -.236 128 -.153 81 205 -.517 -.347 175 
32 355 -.109 -.498 278 82 -.362 -.456 -.289 193 
33 -.255 -.042 -.621 039 83 -.315 -.112 -.450 388 
34 519 -.331 -.017 .288 84 -.050 -.007 -.654 -.159 
35 214 .095 621 -.150 85 041 599 -.307 .069 
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36 .218 .189 488 375 86 -.041 -.094 -.060 -.669 
37 -.104 -.206 247 -.590 87 .000 -.336 -.192 562 
38 294 .049 -.360 -.498 88 125 659 050 .067 
39 537 -.140 355 158 89 -.018 384 365 424 
40 302 528 .293 -.013 90 .030 -.285 -.610 .037 
41 -.245 -.356 358 -.380 91 -.270 -.366 -.498 045 
42 359 406 .253 318 92 -.606 -.112 242 127 
43 596 217 107 214 93 -.536 228 .204 .276 
44 356 052 .049 574 94 -.455 314 300 -.254 
45 .560 -.165 -.341 -.069 95 -.348 340 321 342 
46 .216 -.412 17 -.249 96 -.511 -.352 212 .163 
47 496 124 -.205 395 97 -.294 -.131 -.050 599 
48 .273 -.358 121 -.49] 98 -.605 .196 198 -.117 
49 -.276 -.522 282 -.179 99 -.321 -.078 487 337 
50 .286 .096 -.076 -.606 


In addition to identifying key dimensions, MDS can also be used to identify which clusters fall within each 
dimension (Seok, 2009). Clusters are one or more unique elements of a whole dimension recognized as being 
equally related (Rogers, 2003; Vishwanath & Chen, 2006). This additional analysis revealed 11 clusters across the 
four dimensions. Table 2 shows the selected items in each dimension along with their coordination from Table 1 and 
clusters with final coordinates. 


Table 2 


Final Positioning of Items in Each Dimension and Cluster 


Dimension |: Dependability and Support 
The degree to which the AT device is reliable, and the level of support offered by maintenance personnel to 
keep it working properly (Batavia & Hammer, 1990; Kajko-Mattsson, Karim, & Mirjamdotter, 2011). 


Cluster of AT Care 
4 Values of Items 
Configuration 
74 ~.664 The AT device needs constant adjustment and/or excessive care in everyday use. 
A special room/environment is required (e.g., heating, cooling, dust free, low or high 
43.596 oo 
humidity). 
61 564 The instruction book spells out all maintenance routines to be followed. 
Problems can arise if the equipment is not operated according to prescribed operating 
45 .560 : : 
instructions. 
39 537 The device has special design features (e.g., plug-in modules) that can reduce the difficulty 
; of repairs. 
57.532 Routine maintenance is necessary. 
47 406 Care and maintenance are necessary for the AT device to last throughout (and beyond) its 
, expected life. 
59.465 The maintenance record forms are adequate. 
58 395 Maintenance record forms are provided. 
3 371 I have access to somebody whom I can consult with regarding AT. 
77 «365 The turn-on/start-up routines must be followed each time the AT device is used. 
30.361 The likely cost of repair is affordable. 
40.302 Spares have been provided for repair. 


Cluster of AT Functionality 


154 


Values of Items 


Configuration 
31 = .603 The AT device is covered by public or private insurance (or other financing programs). 
34.519 The AT device must interface with other devices. 
65.384 The device functions as claimed by the manufacturer. 
75 372 There are indications that the equipment is ready to use (e.g., meter readings, lights on or 
off, signals). 
42 359 These types of devices have a prior breakdown history. 
32 = 355 There are warranties on the AT device. 
50.286 A technician or engineer is required for initial assembly or installation. 
Cluster of External Reliance 
4 Values of Items 
Configuration 
53.476 Special tools are required for assembly, installation, or start-up. 
48.273 The supplier will assemble and/or install the AT device. 
Cluster of Team Participation 
4 Values of Items 
Configuration 
10. 243 The use of AT devices and services can be monitored by the AT team to ensure successful 
implementation. 
2 139 I feel confident in my ability to participate in an AT assessment. 


Dimension 2: Device Features 
The characteristics that affect compatibility between the user, the AT device, and academic success (Mirenda, 
2001; Monk et al., 2006; Raskind & Higgins, 1998). 


Cluster of Device-User Compatibility 


4 Values of Items 
Configuration 
88  .659 The AT device depends upon an external power supply or other hook-up. 
85.599 The AT device has safety features (e.g., emergency brakes). 
12 563 The visuals of the AT device look fine. 
51.558 Instructions for assembly and/or installation are included in the manual. 
54. $49 Test equipment (e.g., computer, multimeter, oscilloscope) is required for start-up or 
calibration. 
ll 475 The information is presented in an effective way (including use of speech, music, graphics, 
text, and animation). 
24 = .380 The visual distance is appropriate for the student. 
28 = .332 The price of the AT device is appropriate. 
Cluster of Academic-Related Features 
4 Values of Items 
Configuration 
22 ~=~«.621 The student has keyboarding proficiency (if applicable). 
20.564 The student can learn to use the AT device during the orientation. 
17 456 The student sustain attention using the AT device. 
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15.453 The AT device relates to my school’s curriculum. 


67 = .392 To meet the student’s specific needs, other important needs are ignored, such as academics. 
9 358 The AT device and services can help students achieve their IEP goals. 

320 I feel confident in my ability to assess students’ need for assistive technology (AT). 
94 «314 The AT device motivates the student’s learning. 


Dimension 3: Ease of Use 
The degree to which the AT device is simple and convenient to operate over time (de Joode, van Heugten, 
Verhey, & van Boxtel, 2010; Demers, Weiss-Lambrou, & Ska, 2002). 


Cluster of User Friendliness 


4 Values of Items 
Configuration 
56.672 Maintenance is easily handled by the student (or personal assistant). 
19 .581 The AT device can be used by the student for multiple tasks. 
70.565 The student can independently go through all start-up and diagnostic routines. 
62 .562 The instructions are effective. 
21 = .554 The student has enough overall technology literacy to use the device. 
52530 The instructions are complete, concise, clear, and easy to follow (i.e., a logical step-by-step 
procedure). 


Cluster of AT Convenience 


4 Values of Items 

Configuration 
27 ~—.506 The AT device demonstrates ease of portability across settings. 
36 =—.488 It is reasonable to expect me to carry out some of the repairs. 
99 487 The AT device helps my students become independent. 
37.247 Some types of repairs must be performed by an expert. 
Cluster of Long-Term Use 
4 Values of Items 

Configuration 
35621 The AT device is likely to become obsolete soon due to compatibility problems with 

devices now being developed. 

29 429 The likely cost of maintenance is affordable. 
46 Al7 The AT device remains dependable under repeated use. 
41.358 The device is dependable. 
26 =.144 The AT device is appropriate for future use. 


Dimension 4: Efficiency 
The effectiveness of the AT device in helping the user meet his or her challenges as well as the user’s comfort 
level with the device (Fuhrer, 2007). 


Cluster of User Success 


4 Values of Item 

Configuration 
5 .677 I am aware of my students’ specific academic needs. 
97.599 The AT device helps my students learn how to learn. 
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95.342 The interface of the AT device is efficient for my students. 


4 333 I can help my students achieve academically using technology. 
93.276 The AT device meets the students’ academic needs. 
8 .256 The AT device can help students access the general education curriculum. 


Cluster of Operational Features 


Values of Item 
Configuration 

64.635 The manufacturer addresses the function of the device. 

16 $70 Certain tests or readjustments must be made when the equipment is used during the initial 
, warm-up/use phase. 

60.537 Operation and maintenance manuals are included with the AT device. 

89.424 The AT device can be adapted to hook up in different locations. 

83 388 The AT device has special features to enhance comfort (e.g., a special seating system or 


shock absorbers in the case of a wheelchair). 


Figure | depicts each item of the four MDS models mapped in the MDS space. The dimensions and 
clusters suggest that the validated measure could contribute to the design of conceptual models that may be used to 
help enhance the quality of AT outcomes. 


Dimension 3. Dimension 4 


Dimension 2 


Dimension 1 


Dimension 1 Dimension 2. Dimension3 Dimension 4 


Figure 1. Object points in common space. 
Limitations 


The findings should be interpreted with care. While the small sample size is not believed to have had a 
bearing on the MDS analysis, sampling bias is a concern, in that the participants may not be representative of special 
education professors, thereby warranting caution in generalizing the findings. There is also a concern regarding the 
items themselves, as technology advancements can quickly lose their relevance, to include AT. Consequently, the 
items should be reexamined prior to their use, in the context of benefiting today’s youth. Furthermore, the AT 
practices for outcomes assessment primarily focused on the classroom setting; other locales should be examined, to 
include rehabilitation centers, hospitals, and individual homes to gain additional viewpoints for assessing the quality 
and outcomes of AT. 
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Conclusion 


The need for valid measurement of AT outcomes as a quality indicator in special education has been widely 
expressed. However, outcome assessments associated with quality AT have fallen short of satisfactory validation. 
Furthermore, the existing research has examined AT from the perspective of the AT user, neglecting other important 
points of view, such as educators and preservice teachers at the secondary level. Toward this goal, the present study 
validated items for AT evaluation, identifying dimensions that underlie AT practices along with quality of AT 
outcomes from the perspective of special education postsecondary teachers. Using MDS as the statistical approach, 
four participants rated 19,404 pairwise comparisons, with the results revealing four underlying dimensions and 11 
clusters across the dimensions. While more research is warranted, these dimensions and clusters may be used as a 
foundation for the development of an assessment instrument to enhance the quality of AT outcomes for students 
with disabilities. 
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Descriptors: e-learning and medical education 
Abstract 


A literature review study was conducted to analyze the effectiveness of e-learning to train or educate either 
physicians or medical students on a new skill or knowledge in a healthcare setting. Sixteen relevant research articles 
were identified and retrieved through different databases and were later analyzed. It was found that e-learning was 
employed to teach various healthcare topics, however, palliative care and infection control topics were most popular. 
e-learning was mostly employed to improve learners’ knowledge on a particular topic. The effectiveness of e- 
learning to improve knowledge and skills acquisition varied, however, it was found to be at least as effective as 
traditional instructional method. Hence, the effects of e-learning to improve knowledge or skills acquisition may be 
contingent to the setting, subject matter, and the delivery method. Therefore, before blindly employing e-learning to 
medical education, it is imperative to assess its effectiveness to teach the specific knowledge or skill. 


Introduction 


The present medical curriculum has reached its capacity and it is not viable to remove some of the present 
content to make room for the new information. Additionally, increasing students required hours of instruction is not 
practically feasible (Ozuah, 2002). Along with this challenge, the field of medical education is also facing loss of 
clinical revenues. Therefore, in such situations providing information in a traditional setting is not plausible. By 
keeping the busy schedules of the physicians and medical students in mind, medical educators and trainers may opt 
to use non-traditional options to provide instruction that they can access at their convenience. Distance education is 
one such method that can be used to provide instruction to the learners to help educate them and reduce the gap in 
knowledge. This is an economical option and no changes to the current medical curriculum is required. 


What is e-learning? 


e-learning which is also known as online learning, web-based learning, distributed learning, computer 
assisted or internet-based learning (Ruiz, Mintzer, & Leipzig, 2006). The term e-learning now incorporate distance 
learning and computer-assisted instructions as internet becomes the integrating technology for e-learning. e-learning 
is defined as “learning conducted via electronic media, typically on the Internet” (“E-Learning,” n.d.). e-learning is 
not merely a delivering agent or a broadcast of information through the medium of internet. It is rather a pedagogical 
approach that involves instructions that are learner-centered, flexible, and engaging for the learners. e-learning 
instructions also stimulate communication, collaboration, and interaction. This type of instruction is often used in 
asynchronous environments but it could be used in other environments as well (Ellaway & Masters, 2008). 
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Advantages of e-learning in Medical Education 


There are some advantages of using e-learning compared to the traditional face-to-face instructions in medical 
education. Both educators and learners can benefit from the development of a good e-learning course. The following 
are some advantages of using e-learning instructions in medical education: 


e-learning is less expensive to deliver instructions compared to the face-to-face setting. That is, once the 
educators have developed an e-learning course on a given medical education topic, they are able to 
simultaneously administer the course to a vast number of medical students around the world compared to 
that in the face-to-face setting. 

e-learning provides consistent content to all learners. That is, all medical students receive same 
homogeneous instructions and training on a particular topic. This aids in creating a congruent medical 
educational system. In a face-to-face setting, the instructional content that is delivered to the medical 
students usually depends upon the instructor who is teaching the course. Different instructors may teach 
different information for the same topic. This may lead to medical students having dissimilar access to the 
instructional information or materials and thus, may create an inconsistent medical education system. 
Instructor can easily update the e-learning content. That is, if there is a situation where new medical 
procedures are developed and they need to be replaced with the older ones listed in the medical education 
curriculum, then the instructor of educator could easily update this information in an e-learning course. e- 
learning courses are self-paced. This provides learners (i.e. medical students) the autonomy to access the e- 
learning courses at any time. Students have control over their learning process as they can customize the 
learning materials according to their needs and can take as much time as they need to complete the 
instructions. 

e-learning is a faster method of teaching and learning than face-to-face instruction. Learners (i.e. medical 
students) can skip the materials that they already know to save the instructional and learning time. 
e-learning could also be used to help improve learners’ retention by employing various types of content 
(1.e. images, sounds, and text). It is said to improve retention by incorporating many elements such as 
quizzes, audios, videos, and interactions among others to reinforce the message to the learners. Medical 
students can therefore learn from 

e-learning provides immediate feedback, and encourages interaction with other e-learners and e-instructors 
by using features such as chat room, discussion boards, and email among others (Cantoni et al., 2004). 
Medical students can receive immediate feedback on the practice activities and on the final assessments 
that are built into the e-learning course to foster learning. 


Purpose of this Study 


Despite the abovementioned advantages of employing e-learning instructions, there is still some uncertainty 


regarding its effectiveness on medical students’ and physicians’ learning. In the field of healthcare, erring is too 
costly. Therefore, it is extremely important for the medical students and physicians to have the appropriate level of 
skills and knowledge to practice medicine as they are responsible for human lives. Using e-learning to train medical 
students and physicians may be a faster and cheaper method to provide instructions. However, it is important to 
assess if e-learning is also effective in promoting learning in the healthcare setting. This is because, effectiveness 
plays an important role in learning. Instructions may be easy to use and economical to deliver but if it is not 
effective in educating or training the learners (i.e. medical students or physicians), then they are considered to be 
substandard instructions. Despite the importance of knowing the extent of effectiveness of e-learning in medical 
education, there is a paucity of research conducted on this topic. 


To address this uncertainty, it is important to evaluate the use and effectiveness of e-learning in a particular 


setting and compare the results within that setting. Hence, this study focuses on the uses and effects of e-learning in 
healthcare. Previously, there have been research studies that have employed e-learning in the healthcare field to train 
healthcare personnel such as nurses, physicians, medical students, healthcare workers, and volunteers among others. 
However, in this study, effects of e-learning on medical students’ and physicians’ knowledge or skill acquisition will 
be assessed. This analysis will help us understand the effectiveness of e-learning in fostering knowledge or skill 
acquisition among medical students and physicians. The results of this study will guide future research and will 
provide suggestions for the use of e-learning in training and educating medical students and physicians. 
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Research Questions 


The goal of this study was to analyze the research on the use of e-learning to train or educate either 
physicians or medical students on a new skill or knowledge. The following research questions guided the study: 
e What subjects were taught in healthcare using e-learning? 
e How has e-learning been used in healthcare to train medical students or physicians? 
e To what extent, was e-learning effective in fostering medical students or physicians to acquire the intended 
knowledge or skills? 


Method 
Selection Criteria 


A literature review study was conducted to answer the abovementioned research questions. This study did 
not involve human subjects and thus, no internal review board (IRB) approval was sought for this study. Researchers 
established and adhered to some stringent selection criteria throughout the study. The research studies that focused 
only on e-learning in healthcare setting were selected for this study. The population of interest in these research 
studies were either medical students or physicians. Therefore, studies focusing on healthcare professionals other 
than medical students or physicians were excluded from this study. Additionally, research studies that focused on 
employing e-learning in government or industry settings were excluded from this study. 

Additionally, it was required that the articles assessed the effectiveness of e-learning on learning and 
collected empirical qualitative or quantitative data supporting their claims. Research studies that employed either 
data extraction or experimentation method to collect data were included in this study. Review articles, or articles that 
presented authors’ personal opinions or perspectives were excluded from this study. 


Identification of Eligible Studies 


Relevant research articles were identified and retrieved through a several search attempts across different 
databases. First, a general search was conducted using Google Scholar. Following this, additional searches were 
conducted in PubMed, ERIC, and Academic Search Complete databases to find relevant articles for this study. The 
following key terms were employed to retrieve suitable reference articles: “eLearning medical’, “eLearning in 
medical education”, “eLearning physician”, “eLearning medical student”, “eLearning in CME”, and “eLearning 
medical rotations”. This method generated a list of numerous articles. However, only 16 peer-reviewed articles met 
the abovementioned selection criteria set by the researcher. 

The selected articles employed e-learning either in the form of independent or blended learning to teach a 
particular skill or knowledge to either medical students or physicians. These articles showcased the studies that 
employed e-learning in different disciplines within the healthcare setting. They also provided empirical evidences to 
support their claims and provided evidence on the effectiveness of e-learning. Studies with empirical data were 
chosen because relying on empirical sources would help make fewer errors when reporting and making inferences 
regarding the effectiveness of e-learning. 


Analysis of the Study 

The published studies were analyzed to identify the common themes for the uses and the effects of e- 
learning in educating and training medical students or physicians on healthcare topics. The studies were analyzed for 
the following characteristics: topic of instruction, population of interest, sample size, learning objective, type of e- 
learning, effectiveness of e-learning, and learners’ perception towards e-learning. The results of the analysis are 
discussed in the following paragraphs. 

Results 

Topic of Instruction 

A review of 16 papers indicated that e-learning has been used in different healthcare domains to teach 


different topics. e-learning was used to teach the following topics: palpation and ultrasound imaging of the knee, 
immunology, prescribing education, pediatric emergency medicine, how to use the CPAx, breaking bad news, core 
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communication, and epidemiologic and social topics in medicine (such as discussions around palliative care & 
medical test utility), endocrine pharmacology, infection control and congenital infections, healthcare-associated 
infection prevention and control, physiotherapy skills training, dermatology, occupational medicine (i.e. maternity 
protection, hepatitis B in a medical student & occupational asthma), palliative care, screening and brief intervention 
(SBI) for unhealthy alcohol use, and radiological anatomy. Based on the review, it was found that there were a 
variety of topics explored by different researchers to investigate the effects of e-learning on learning outcomes. 
However, palliative care, and infection control are topics that were more popular than the others. 


Participants 


Population of interest. Among the 16 reviewed articles, majority of the studies (n = 12) had medical 
students as their population of interest. There were four studies that had either trainee doctors, clinicians, residents, 
or interns as their population of interest (see Table 1). Medical students in these studies ranged from first year 
students to third year students. Six out of 12 studies had second year students as their population of interest. There 
was only one study that had both second year and third year students as their population of interest (see Table 2). 

Sample Size. The sample size of the 16 studies varied substantially. The sample size of participants ranged 
from 16 to 688. Among these studies, there were 10 studies that had a sample size of more than 100 participants, 3 
studies had participants between 50 and 100, and 3 studies had less than 50 (see Table 3). 


Table 1. Population of Interest in the Reviewed Articles 


Population of interest n Studies 

Physicians (including 4 Brooks et al (2016); Chang et al (2014); Corer, Handy, & Brett (2016); 
residents, clinicians, Daetwyler, Cohen, Gracely, & Novack (2010) 

and interns) 

Medical Students 12 Arroyo-Morales et al (2012); Gaikwad & Tankhiwale (2014); Silva, Souza, 


Filho, Medeiros, & Criado (2011); Khasawneh, Simonsen, Snowden, 
Heggins, & Beck (2016); Preston et al (2012); Boye, Moen, & Vik (2012); 
Day et al (2015); Smits et al (2012); Tan, Ross, & Duerksen (2013); Webb 
& Choi (2014); O’Neill et al (2011); Truncali et al (2011) 


Table 2. Medical Students’ Year in Medical School 
Year in Medical School n Studies 
First Year Students 2 Webb & Choi (2014); Truncali et al (2011) 
Second Year Students 6 Arroyo-Morales et al (2012); Boye, Moen, & Vik (2012); Gaikwad & 


Tankhiwale (2014); Silva, Souza, Filho, Medeiros, & Criado (2011); Smits 
et al (2012); Preston et al (2012) 


Third Year Students 3. Day et al (2015); Khasawneh, Simonsen, Snowden, Heggins, & Beck 
(2016); Tan, Ross, & Duerksen (2013) 


Second & Third Year 1 O’Neill et al (2011) 


Students 
Table 3. Reported Sample Sizes in the Reviewed Studies 
Sample n Studies 
size 
<50 3. Arroyo-Morales et al (2012); Gaikwad & Tankhiwale (2014); Silva, Souza, Filho, 
Medeiros, & Criado (2011) 
51-100 3 Daetwyler, Cohen, Gracely, & Novack (2010); Khasawneh, Simonsen, Snowden, 


Heggins, & Beck (2016); Preston et al (2012) 


101-150 5 Boye, Moen, & Vik (2012); Day et al (2015); Smits et al (2012); Tan, Ross, & Duerksen 
(2013); Webb & Choi (2014) 
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> 150 5 Brooks et al (2016); Chang et al (2014); Corner, Handy, & Brett (2016); O’Neill et al 
(2011); Truncali et al (2011) 


Learning Objectives 


The studies were further categorized by their learning objectives. Three types of learning objectives namely 
knowledge, skills, and both (i.e. knowledge & skill) were identified for this study. Among the 16 studies, majority of 
the studies (n = 10) used e-learning to only improve learners’ knowledge on a particular topic. There were only two 
studies that used e-learning to improve only the learners’ skills, and there were four studies that used e-learning to 
provide knowledge and skills training to the learners (see Table 4). 


Type of e-learning 


Among the 16 studies, there were seven studies that incorporated e-learning as part of the curriculum, with 
lectures or practical tutoring (see Table 5). This type of e-learning is known as blended e-learning. The remaining 
studies employed e-learning as an independent resource to improve learners’ skills or knowledge. This is known as 
independent e-learning. All studies focusing on improving knowledge or skills employed a didactic form of e- 
learning that involves employing web-based modules to provide instructions to the learners. Whereas, some studies 
focusing on improving skills employed didactic e-learning instructions along with other instructional strategies. For 
example, Webb and Choi (2014) employed virtual interactions (i.e. virtual patients in a clinical scenario). Whereas, 
some researchers embedded demonstrative videos into the e-learning modules. Preston et al. (2012) employed 
video-clips of patient therapist simulations, where Corner, Handy, and Brett (2016) employed video case studies of 
CPAx assessments in to the modules. On the other hand, Daetwyler, Cohen, Gracely, and Novack (2010) used a 
form of a role play method to provide learners the opportunity to practice communicating bad news to the 
standardized patients in a virtual environment (i.e. using video-conference). 


Table 4. Review Articles Categorized by Learning Objectives 


Learning : 
Objectives é pmdies 
Knowledge 10 Brooks et al (2016); Chang et al (2014); Boye, Moen, & Vik (2012); Day et al 
(2015); Smits et al (2012); Gaikwad & Tankhiwale (2014); Silva, Souza, Filho, 
Medeiros, & Criado (2011); Khasawneh, Simonsen, Snowden, Heggins, & Beck 
(2016); O’Neill et al (2011); Tan, Ross, & Duerksen (2013) 
Skills 2 Corner, Handy, & Brett (2016); Preston et al (2012) 


Both (Knowledge 4 Arroyo-Morales et al (2012); Daetwyler, Cohen, Gracely, & Novack (2010); Webb 
& Skills) & Choi (2014); Truncali et al (2011) 


Table 5. Type of e-learning Used in the Review Articles 
Type of e-learning n Studies 


Blended 7 Arroyo-Morales et al (2012); Boye, Moen, & Vik (2012); Daetwyler, Cohen, 
Gracely, & Novack (2010); O’Neill et al (2011); Preston et al (2012); Silva, 
Souza, Filho, Medeiros, & Criado (2011); Webb & Choi (2014) 


Independent 9 Brooks et al (2016); Chang et al (2014); Corner, Handy, & Brett (2016); Day et 
al (2015); Smits et al (2012); Gaikwad & Tankhiwale (2014); Khasawneh, 
Simonsen, Snowden, Heggins, & Beck (2016); Tan, Ross, & Duerksen (2013); 
Truncali et al (2011) 


Effectiveness of e-learning 


When designing e-learning instructions, Instructional designers, instructors, and e-learning developers have 
a learning outcome in mind that they want to accomplish. Their aim is to develop instructions and use technologies 
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to create learning environments that are effective in fostering learning. In this section, the effectiveness of e-learning 
will be discussed. The effectiveness of e-learning is marked by its ability to achieve statistically significant effect on 
improving learners’ knowledge or skills. Thirteen out of 16 studies, found that e-learning is effective in improving 
either knowledge or skill acquisition among medical students and physicians. 

Knowledge. Among the studies that investigated the effects of e-learning on knowledge (n = 10), seven 
studies reported that there was a statistically significant effect of e-learning on the learners’ knowledge. Several 
researchers have assessed pre-test and post-test scores to measure the effectiveness of e-learning modules or courses 
on the learners’ knowledge. The researchers found that there was statistically significant improvement in learners’ 
knowledge. The use of asynchronous e-learning modules led to an increase in learners’ post-test scores (Boye, 
Moen, & Vik, 2012; Brooks et al., 2016; Chang et al., 2014; Gaikwad & Tankhiwale, 2014; O’Neill et al., 2011; 
Silva, Souza, Filho, Medeiros, & Criado, 2011; Tan, Ross, & Duerksen, 2013). 

Tan, Ross, and Duerksen (2013) reported that including virtual patient (VP) cases in palliative care 
curriculum and allowing students to interact with the VP helped students gain knowledge about the topic. The 
results indicated that there was a significant increase in the post-test knowledge scores from the pre-test scores. 
Boye, Moen, and Vik (2012) reported that effects of e-learning vary. The e-learning module in this study provided 
coaching on immunology. It was found that student who used the e-learning modules had higher scores on the 
immunology test than students who did not use the e-learning module. Additionally, the researcher found that 
intermediately skilled students benefitted from the e-learning modules than the less skilled or best skilled students. 
The results indicate that e-learning could be used as a supplement to the traditional teaching to improve learning 
(Boye, Moen, & Vik, 2012). 

Brooks et al. (2016) reported that trainees who spent less time completing the online modules had low pre- 
test scores on average and had inadequate improvement in their knowledge on the post-test scores compared to the 
trainees who spent more time completing the modules. This shows that the effectiveness of the e-learning module on 
improving learners’ knowledge is dependent on how the learner interacts with the module and how much time they 
spend completing the modules. Therefore, along with creating good e-learning instructions, it is important to also 
create rules and regulations and monitor its use (Brooks et al., 2016). 

The abovementioned studies found that e-learning was overall successful in improving learners’ knowledge 
in different disciplines. However, contradicting results were reported by some researchers as they found no 
significant gain in knowledge from using e-learning modules. Day et al. (2015) found that there was no significant 
difference in students’ knowledge after the doctoring sessions as assessed by the pre-test and post-test scores. 
However, there is a non-statistical trend showing that the students who completed eDoctoring session had more 
number of correct answers on the test compared to the students who completed the traditional small group doctoring 
session. Khasawneh, Simonsen, Snowden, Heggins, and Beck (2016) found that assessing e-modules did not 
improve students’ knowledge and post-test scores. Similar results were reported by Smits et al. (2012). The 
researchers found that there was no difference in students’ knowledge when they received case-based e-learning or 
text-based learning instructions. Majority of the studies found significant effect of e-learning on knowledge 
acquisition. However, there were some studies that did not find a significant effect of e-learning on knowledge 
acquisition. These results indicated that e-modules is not a widely successful instructional tool (Khasawneh et al., 
2016). 

Skills. Conducting the review of the articles revealed that e-learning is also effective in teaching skills to 
the learners. Among the 16 articles, there were two studies that assessed the effectiveness of e-learning on skill 
acquisition. Corner, Handy, and Brett (2016) used e-learning to teach clinicians how to use the Chelsea Critical Care 
Physical Assessment (CPAx) tool. The researchers found that after completing the e-learning modules, clinician felt 
that they could use the CPAx tool consistently and could also explain their colleagues how the CPAx score works. e- 
learning was also found to be effective in teaching physiotherapy skills to the medical students. Preston et al. (2012) 
reported that students who used the Physiotherapy eSkills Training Online resource scored higher on the practical 
exam than the students who only had access to the usual traditional teaching. This shows that e-learning can be an 
effective tool to teach practical skills to the learners. 

Both knowledge and skills. Among the reviewed articles, four studies investigated the effects of e-learning 
on knowledge and skills simultaneously (Arroyo-Morales et al., 2012; Daetwyler, Cohen, Gracely, & Novack, 2010; 
Truncali et al., 2011;; Webb & Choi, 2014). Daetwyler et al. (2010) studied the educational interventions to improve 
interns’ ability to communicate and break bad news to patients. In this study the researchers compared the use of 
didactic e-learning to the didactic and web encounter exercise based e-learning in teaching interns the necessary 
medical communication knowledge and skills. The results of the two-type of e-learning were compared to the scores 
of the interns’ who were in the control group who did not receive any e-learning intervention. The researchers found 
that the interns in the e-learning groups performed better at communicating bad news to patients than the interns in 
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the control group. They also found that interns in the didactic and web encounter exercise based e-learning group 
outperformed the interns in the other group (i.e. control and didactic e-learning) in communicating bad news to 
patients during a practical assessment. 

This shows that adding an online practice component to e-learning improved its effectiveness in enhancing 
the interns’ ability to effectively communicate bad news (Daetwyler et al., 2010). All the studies except Arroyo- 
Morales et al. (2012) found that their e-learning modules were effective in improving both knowledge and skills 
among the learners. The researchers found that the e-learning module was effective in improving learners’ palpation 
and ultrasound musculoskeletal examination skills. However, it was not effective in helping learners acquire 
knowledge. 


Learners’ Perception Towards e-learning 


Along with assessing the effectiveness of e-learning instructions, is important to consider how learners 
perceive the e-learning instructions. Specifically, it is imperative to assess students’ perceived satisfaction with and 
usability of e-learning. After reviewing the articles, it is found that overall students perceived e-learning modules as 
easy to use and had no technical difficulties (Truncali et al., 2011). Most of the clinicians (i.e. 93.6%, n = 264) who 
used the e-learning module to learn the CPAx tool reported that the module was simple to use (Corner, Handy, & 
Brett, 2016). 

Additionally, the students perceived the e-learning modules to be useful for learning practical skills 
(Preston et al., 2012). Gaikwad and Tankhiwale (2014) found that the students accepted the e-learning activity and 
perceived it as “innovative, convenient, flexible, and useful” (p. 16). It was found that the students largely had good 
impression about the e-learning program. When students were asked how well did the e-learning course serve as a 
supplement to the regular teaching, 70% of the students reported it being good or very good (Boye, Moen, & Vik, 
2012). Khasawneh et al. (2016) reported that approximately 70% of the students who completed the e-learning 
module on infection control and 72% of the students who completed the e-learning module on congenital infection 
some satisfaction or good satisfaction for using e-learning modules as learning tool. Similarly, Silva et al. (2011) 
also found that 90.9% of the students reported having good to excellent satisfaction with the online course (i.e. e- 
learning course). 


Discussion 
Conclusion 


Medical students and physicians go through intensive medical training and education. They have busy 
schedules and are required to acquire specific knowledge and skills to complete their education or to continue their 
practice. In addition to the regular curriculum, practitioners have identified new skills and knowledge that need to be 
taught to the medical students and physicians in order for them to be up to date and competitive in the field. 
Asynchronous e-learning and blended e-learning are being used to provide instructions to the learners. 

After reviewing the 16 identified articles, it is found that didactic e-learning courses were useful in 
providing instructions. Asynchronous and blended methods of e-learning were effective in improving learners’ 
knowledge and skill acquisition. However, effectiveness may vary based on setting and discipline. Some studies 
reported no significant difference between e-learning and traditional instructions when measuring learners’ 
knowledge and skills after the interventions. Whereas some researchers reported that there was a significant effect of 
e-learning on knowledge and skill acquisition. Therefore, there is no clear consensus on the effectiveness of e- 
learning as its effectiveness is highly contingent on the learner population, topic of instruction, design of e-learning, 
and the setting that it is employed in. 

However, no study has reported adverse effects of e-learning on learners’ knowledge and skill acquisition. 
Researchers have found that e-learning is at least as effective as traditional learning in enhancing knowledge and 
skill acquisition. Therefore, e-learning is not just a fad. It is an effective method to educate and train medical 
students and physicians on topics that are within and outside the curriculum in an economical and sustainable 
manner. To make e-learning effective, it is suggested that e-learning should be added as a supplement to the 
traditional teaching and must not replace traditional learning (Arroyo-Morales et al., 2012). 
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Limitations 


This study was conducted to understand the effectiveness of e-learning on medical students’ and 
physicians’ knowledge and skills acquisition. Different types of research have been conducted in these studies. 
Researchers employed a randomized control group experiment, quasi-randomized control group experiment, and 
data gathering techniques, among others to measure the effectiveness of e-learning on learners’ knowledge and skills 
acquisition. However, they all yielded more or less similar results regarding the effectiveness of e-learning. 

There were differences in the studies’ population, sample size, setting, and topic of e-learning. This makes 
it difficult to directly compare the results between studies and to make inferences from these studies. Additionally, 
some articles had unequal sample size in the groups being compared. This may have led researchers to make type I 
or type II errors when making comparisons and thus, may have affected the results. Hence, we must be cautious 
when making inferences based on such results. 


Suggestions for Future Research 


e-learning is being used as educational intervention in medical education. However, the benefits of e- 
learning may be contingent to the setting, subject matter, and delivery method. Therefore, it is important to be 
cautious about using online learning in the medical education. Before blindly employing e-learning in medical 
education, it is imperative to know if e-learning would be an appropriate method to teach the specific knowledge or 
skill and if it is effective in doing so. Because if even the online instructions are well designed, easy to use, and 
efficient but if they lack effectiveness in teaching a particular knowledge or skill, then it is worthless to use them. 
Therefore, it is important to evaluate the effectiveness of the instructions before employing it. 

Since there is no consensus on the effectiveness of e-learning in healthcare, future research studies are 
warranted to evaluate the impact of e-learning in each healthcare setting to understand the effects and usefulness of 
the e-learning educational approach (Khasawneh et al., 2016). Most of the studies have compared the e-learning 
with traditional face-to-face instructions. From these studies we learned that e-learning is at least as effective as 
traditional instructions. However, there is a need to stop comparing e-learning (i.e. distance education) to the 
traditional instructions. Rather, we must shift our focus to evaluate the different components of e-learning and the 
different instructional strategies used in developing e-learning instructions. 

It is important to study how instructional strategies, technologies, and learning theories can be employed to 
create effective e-learning environments. Conducting such studies will help us understand what strategies work best 
or less in asynchronous e-learning environments and how we can create effective learning experiences for our 
learners. Future researchers must consider conducting randomized experimental studies to evaluate the effectiveness 
of e-learning. This will help them control for confounding and extraneous variables and will help them identify the 
exact impacts and effects of e-learning on learners’ knowledge and skills acquisition. 
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Abstract 


In today’s information-rich world, digital literacy includes the ability to quickly evaluate social media for 
disinformation. Improving digital literacy involves teaching social media users how to verify posts, but how do we 
motivate users to actually do this evaluation in their daily lives? Video games may be one way. This article presents 
research on a social-impact game, Fake It to Make It, which positions players as for-profit disinformation 
disseminators. Drawing upon the BJ Fogg’s Functional Triad for Persuasive Computers and paying particular 
attention to the usability and perceived credibility of Fake It to Make It, this research analyzed the game from a 
persuasive design lens using player-participant data. This was accomplished through screen-captured gameplay as 
well as interviews and retrospective think-alouds. Additionally, to determine if the game affects abilities to assess 
claims on social media, pre- and post-intervention media literacy assessments were utilized. With this data, the 
researchers provide design recommendations to increase usability, influence procedural knowledge on social media, 
and promote continued gameplay and greater emotional/behavioral impact. 


Introduction 


How can we motivate social media users to critically analyze potential disinformation? Video games may 
be one way. This project presents research on a fake news simulator created by designer/developer Amanda Warner, 
Fake It to Make It, which positions players as for-profit disinformation disseminators. Specifically, this research 
includes: (1) an analysis of the game from a persuasive design lens using player-participant data, (2) an analysis of 
participant abilities to assess claims on social media before and after exposure to Fake It to Make It, and (3) design 
recommendations for greater emotional/behavioral impact and usability. 


Problem 


A concerning study by Stanford researchers shows that many college students lack the skills to distinguish 
between legitimate journalism and false, misleading, or satirical “news” (Wineberg, 2016). In addition to helping 
students spot signs of sensationalized or fictional online “news” articles, it is key that they also recognize how it is 
spread and its social impact. Yet, how can we motivate social media users to critically analyze potential 
disinformation when it is easier to simply allow confirming viewpoints into social media feeds? Well-designed 
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video games may be one way to motivate learners to scrutinize news sources in social media spaces. Consequently, 
it is crucial to analyze the effectiveness of existing serious games as media literacy tools. 


Background 


Fake It to Make It is a self-described “social-impact game about fake news” (Warner, n.d.) This single- 
player, browser-based game positions players as disinformation “entrepreneurs”; players spread illegitimate news— 
using techniques such as purchasing bots and manipulating viewers’ fears, anger, and other strong emotions—in 
order to generate shares, and, thus, profit. The developer of the game, Amanda Warner, states that her hope with the 
game is to see the actions, attitudes, and beliefs of players change after playing the game. (Warner, n.d.) 

Fake It to Make It does indeed provide a space for players to identify common themes and techniques used 
in the dissemination of false or misleading stories. The researchers assert, however, that while Warner’s game was 
created to facilitate role-playing—a key advantage of persuasive technologies—it may lack the persuasive design 
elements to motivate players to continue playing the game, let alone promote greater media literacy (Fogg, 2007, p. 
141). As such, the following research questions were posed: 


Research Questions 


How does Fake It to Make It utilize persuasive design elements to motivate sustained gameplay? 

How does Fake It to Make It’s usability influence its perceived credibility as a persuasive media literacy 
tool? 

3. How does Fake It to Make It’s current design promote procedural knowledge with claims on social media, 
i.e. verifying a post's sources and motivations? 


NOR 


Theoretical Framework 

Although Fake It to Make It may a American News ays 
have some principle elements of an . — 
educational game (such as explanations for : _ 
the spread of disinformation and a clear 
win-state), key features could be redesigned 
for greater skills transfer (procedural 
knowledge) as well as behavioral and 
emotional impact. This research utilizes BJ 
Fogg’s Functional Triad to determine which 
features require improvements to turn this 
simulation into a persuasive game. 

BJ Fogg’s Functional Triad 

This research draws upon BJ 
Fogg’s concept of captology, the study of 
computers as persuasive technologies, and 
the three roles that computers can play in 
the act of persuasion. They may act as 
persuasive tools, persuasive media, or ‘Figure 1: Players watch as in-game social media NPC react to false 
persuasive social actors—known as the articles. 

Functional Triad, also coined by Fogg. 

Persuasive tools are generally more aligned with teaching the user to act based on making the target 
behavior easier; they lead the user through a preset process, condition by reinforcing a target behavior, and use self- 
monitoring among other techniques (Lin, 2016, p. 661). This study, however, did not gather information on the 
game as a persuasive tool. Warner herself states that it is “possible that this game could inspire someone to make 
fake news, but that [she’s] willing to take the risk, because [she] think[s] the potential for positive change in players 
is worth it.” (Warner, n.d.) As such, the researchers did not attempt to determine if the game promotes the spread of 
disinformation. 

This study, instead, was designed to gain insight into the effectiveness of Fake It to Make It as a form of 
persuasive media and as a persuasive social actor. Persuasive media enables people to explore the causal 
relationships between a behavior and its outcome, to rehearse a behavior so as to create a persuasive experience with 
symbolic and sensory information (Lin, 2016, p. 661). As social actors, computers persuade people by rewarding 


Articles 


Plant An Article 
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them with positive feedback, modelling a target behavior or attitude or providing social support. (Lin, 2016, p. 661). 
See Tables 1 & 2 for the specific elements of persuasive media and persuasive actors that were investigated. These 
elements were studied to determine (a) if players are motivated to continue playing the game in the first place, and 
(b) if it has the elements necessary to promote procedural knowledge with claims made on social media. 


Table 1: Computers as Persuasive Media (Fogg, 2007, p. 139; Fogg, 2003, p. 91) 

Cause-and-Effect Simulations * Allow users to explore and experiment 

¢ Show cause-and-effect relationships clearly and quickly 
¢ Persuade without being overly didactic 


Environment Simulations * Can create situations that reward and motivate people for a 
target behavior 

* Allow rehearsal: practicing a target behavior 

* Can control exposure to new or frightening situations 

¢ Facilitate role-playing: adopting another person’s perspective 


Object Simulations * Fit into the context of a person’s normal life 
* Are less dependent on imagination or suspension of disbelief 
¢ Make clear the impact on normal life 


Table 2: Computers as Persuasive Social Actors (Lin, 2016, p. 662) 


Physical Cues Provide visually attractive computing products (interface or 
hardware) 

Language Use Spoken language to persuade users 

Psychological Cues A perceived sense that the computing product has emotions 

Social Interaction Influence users through interaction with them 

Social Roles Act in role of an authority, expert or a trustworthy figure 


An Emphasis on Usability and Credibility 

In addition to drawing upon the Functional Triad, this research investigates the inherent usability and 
perceived credibility of Fake It to Make It. Just as a credible person can influence other people, credible computer 
products also have the power to persuade. (Fogg, 2007, p. 141) Interactive systems often use what Fogg calls 
“microsuasion,” subtle advice to help users solve a problem, such as successfully navigating a system; as such, an 
interface layout and menu options can be a form of advice. (Fogg, 2007, p.141) If Fake It to Make It’s interface is a 
hindrance in performing certain actions, such as completing in-game tasks, its perceived credibility diminishes. 


Methodology 


To determine the efficacy of the game’s motivational affordances and its effect on procedural knowledge 
with social media posts, the researchers used (1) demographic surveys (2) a pre-test media literacy assessments, (3) 
exposure to the intervention, (4) post-test media literacy assessments, and (5) short semi-structured interviews with 
retrospective think-alouds. 


Participants 

There were seven participants in this study, all of whom completed the demographic survey. This survey 
included questions to reveal their familiarity with video games and perceived proximity to disinformation. Questions 
related to disinformation were adapted from the Pew Research Center (2016). This survey also asked participants to 
disclose their age, gender, the platform on which they play video games (console and/or mobile), and genre of video 
games they play. The researchers requested the participants’ self-reported proximity to gaming and disinformation to 
gain insight into their in-game choices during the intervention, attitudes toward the intervention, and procedural 
knowledge of verifying claims on social media. 


Participant Demographics 

Participant age ranged between 19 and 42 years of age (mean = 28.57, SD =8.17). Four participants 
identified as male and three identified as female. There were four participants that play video games regularly (three 
played both console and mobile, one played only mobile) and three participants reported that they do not generally 
play video games. Of the four participants that play video games, there was an assortment of genres, the most 
popular being first-person shooter, action/adventure, educational and real-time strategy. Regarding disinformation, 
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all seven participants reported that they come across news stories about politics and government that are not fully 
accurate. Four participants said they share articles they see on their social media feed without checking the source 
that produced it (three said sometimes, one said hardly ever); three participants said they never share posts before 
verifying. All seven participants said they were confident in their abilities to recognize made-up news stories (two 
said very confident, five said somewhat confident). 


Data Collection and Analysis 


Pre-Test / Post-Test Assessment of News Literacy on Social Media 

This study used an instrument created by the Stanford History Education Group to measure college 
students’ abilities to assess claims on social media; specifically, participants read a tweet and explain in two short 
essays why it might or might not be a useful source information. (Wineburg, 2016) Upon receiving the participants’ 
answers, the researchers labeled their assessments—according to the Stanford rubric—as showing mastery, 
emerging, or beginning in verification skill. 

Although this research on Fake It to Make It was primarily concerned with the HCI elements of persuasive 
design, it was nonetheless important to determine if the game, as it stands, impacts procedural knowledge. As such, 
a similar post-test assessment was completed by participants, albeit with a different tweet. 


Exposure to the Intervention 

Participants were then asked to play the intervention while their in-game actions were recorded using 
Morae screen-capturing software. These recordings were gathered to aid in retrospective think-alouds. The 
researchers chose this technique to correlate (1) patterns of use and user intentions throughout the game and (2) 
flaws in usability due to interface design. 

Patterns of use are especially important in investigating the game as a persuasive media; as a persuasive 
media, a computer can convey either symbolic content (i.e., text, data graphs, icons) or sensory content (i.e., real- 
time video, virtual worlds, simulation) (Fogg, 2007, p. 137). Connecting the retrospective think-alouds with the time 
spent viewing specific symbols garnered valuable inferences on the game’s effectiveness as a persuasive media and 
actor. 

As noted earlier, the perceived credibility of 


the game is especially important in determining its GBB stoveon.ors © are 
persuasive qualities. Observing participants’ patterns me 

while playing Fake It to Make It allowed the New polling shows the @NRA is out of touch 
researchers to pinpoint exact moments/features that with gun owners and their own members 
hinder interface usability. Additionally, the ampr.gs/1Pyw4qg #NRAfail 


retrospective think-aloud garnered information on the 
perceived attractiveness of the visual design—another, 
often the first, indicator of perceived credibility. (Fogg, 
2007, p. 143) 


: : : VOTE FOR A CANDIDATE WHO SUPPORTED 
The retrospective think-alouds occurred in BACKGROUND CHECKS 


conjunction with the post-gameplay semi-structured 
interview. Independent variables were limited by 
placing individual participants in the same controlled — Figure 2: Students are asked why this tweet might and might not 
environment with the same exposure time (30 be a useful source of information in the pre-test assessment. 
minutes) to the intervention. 


Post-Gameplay Semi-Structured Interview and Retrospective Think-Aloud 

This semi-structured interview drew upon BJ Fogg’s Functional Triad and the different roles of persuasive 
technologies. Participants were asked questions meant to determine which elements of the game could be considered 
forms of persuasive media or persuasive social actors, and then determine if/which elements succeeded in their 
respective roles. In addition to interview questions, the researchers conducted retrospective think-alouds, asking 
participants about their gameplay choices and any perceived usability issues. Lastly, this interview was used to gain 
further information on participant emotional response to the intervention and to validate that the game experience 
triggered the necessary qualities of a serious game, such as presence, flow, fun, and engagement. Without these 
elements, the motivation may be limited and the attractiveness of the intervention is diminished. (Deterding, Dixon, 
Khaled, & Nacke, 2011, p.79) 
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With this qualitative data, the researchers conducted a thematic analysis, a method for identifying, 
analyzing and reporting patterns (themes) within data (Braun & Clarke, 2006). The researchers used the 
aforementioned roles of persuasive media and social actors (Tables 1 & 2) as themes for analysis including sub- 
themes for each role: positive response, negative response, and indifferent response. An additional code, usability 
issue, pinpointed moments that causes gameplay hindrances. 


Limitations 


With only seven participants, the findings of this study may not be generalizable. The relatively wide range 
in age and level of education of the participants may also impact the usability for the Stanford tweet assessments. 
With regard to usability, the researchers did not make use of captured Morae data such as exact time spent on 
specific scenes or the amount and location of clicks. Lastly, although players were told that the researchers were 
analyzing the system and not player behaviors, it is possible that the Hawthorne effect may have influenced in-game 
decisions. 


Research Findings 


RQI1: Does Fake It to Make It effectively utilize persuasive design elements to motivate sustained 
gameplay? 


Coding Results 

The thematic analysis was conducted over 2 hours, 8 minutes, and 22 seconds of recorded audio. Each 
interviewed averaged a time of 18 minutes and 20 seconds. After conducting the thematic analysis of interviews and 
retrospective think-alouds, it became apparent that attitudes toward the game varied greatly. In total, there were 58 
positive codes, 69 negative codes, and 11 indifferent codes in response to aspects of computers as persuasive media 
and persuasive social actors. Based on the frequency of these codes, Fake It to Make It’s ability to act as a 
persuasive media warranted inspection to fully utilize captology in its gameplay. See Tables 3 & 4 for a breakdown 
of these reactions. 

Of course, the demographics of the seven participants—and their reported video game use—could have 
heavily influenced their positive or negative responses to the game. Just as having more experience engaging in 
exercise may make the motivational features of the fitness devices more salient, exposure to video games may have 
influenced user perceptions. (Rupp, 2018, p.83) For instance, the researchers found that non-game players, on 
average, were likely to have 4.7 more negative codes in their transcripts. Although this study was limited to a small 
sample size, these findings highlight the importance of ensuring that gameplay features are designed for target 
audiences. With Warner’s goal of changing player actions and attitudes, redesigning the tutorial aspects of the 
simulation may aid her non-gamer visitors. (Warner, n.d.) 


Table 3: Computers as Persuasive Media 


Negative Response | Indifferent Response | Total Codes 
Cause-and-Effect Simulation | 20 (44.4%) 23 (51.1%) 2 (4.4%) 


Environmental Simulation 18 (45%) 19 (47.5%) 3 (7.5%) 
Object Simulation 12 (41.4%) 15 (51.7%) 2 (6.8%) 


Table 4: Computers as Persuasive Social Actors 


Negative Response | Indifferent Response | Total Codes 
Language Cues | 4 (50%) 4 (50%) 


Physical Cues | 4 (25%) 8 (50%) 4 (25%) 
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Psychological cues, social interaction, and social roles did not emerge as significant themes. This is most 
likely due to simulation not being overly didactic or emphasizing its moral intentions or authority. 
Analysis of Fake It to Make It as a Form of Persuasive Media 


Cause-and-Effect Simulation. 


s 5 Cause-and-effect You have $18.70 You have 1% of your goal of $1000 
simulations have the ability to Show Expenses 


allow users to explore and 

experiment as well as show cause- 

and-effect relationships quickly BE AllSites = Options/Pause | 4 
(Fogg, 2007, p. 139). Some users 
responded positively to the Goals + Hide Completed Goals 
exploratory aspect of the game, 
mentioning the freedom to choose 
where and how they spread 
disinformation and exploring just 
how inflammatory they could be in 
their gameplay. Many participants, 
however, described feeling Figure 3: Participants felt hindered by the scaffolded goals to learn the game | 
constrained by the sequenced goals = mechanics. 

presented in the game. As one 

participant put it, “I didn’t really feel like I was actually in too much control of my own destiny because the goals 
were so specific... I just felt like I was following directions.” 

With regard to Fake It to Make It’s ability 
to show cause-and-effect relationships quickly, the 
animated social media reactions to planted articles 
received the most positive reactions. There were 
comments such as, “It's satisfying to see that you're 
getting the reaction you wanted and it's satisfying to 
know that you're playing people successfully.” 
Despite the positive responses to the social media Nica world Pacole caer to carn 
animation, there were a number of negative codes about this article and are 
associated with the quickness of showing cause- Spreading its a result. 
and-effect relationships. Many participants reported 
. . ‘ Fi Continue 
issues understanding cause-and-effect with in-game } Continue | 
elements, such as “not fully understanding how to 
meet the objectives.” The first and most apparent 
indication that a player was unable to meet their 
goal comes right after the social media impact 
animation, but it does not indicate why the player Figure 4: Participants relied on language cues from animation to 
was unable to reach their goal. Likewise, one interpret result of planted article. 
participant reported not understanding why their 
articles were successful: “[I] don’t really get any reason why people click on [my] site.” The animation simply says 
that the player was successful without indicating why. Although there is the option to view Additional result details 
located on the right-hand side of the page, not a single participant selected this drop-down feature. They, instead, 
focused on the language cues from the simulated guide. 

Environmental Simulation. 

Simulated environments immerse users with sound and images in order to (1) create situations that reward 
and motivate people for a target behavior, (2) allow rehearsal for a target behavior, and (3) facilitate role-playing or 
adopting another person’s perspective (Fogg, 2007, p. 139). 


Create a site with a credibility rating of at least 30 
Add at least two articles to your site 

Have an article with at least 100 shares 

Have an article that has earned at least $10 


Plant an article that triggers happiness 


\8egaggnag 


Plant an article that triggers fear 


Write and plant an article with a drama score of 20 
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An interesting finding of researching 
this game as an environmental simulation 
surrounds the in-game motivations. ee You have 1% of your goal of $1000 
Participants are introduced to the game by 
creating an avatar and selecting a specific goal 
to strive for. As different players had different 88 All Sites 
proximities to gaming, the researchers 
prompted the player to choose the hardest goal 
in the game—raising $2000 for a car. By Figure 5: The goal progress icon took a diminished role in 
choosing one of the less challenging goals, the the top-right corner of the interface. 
exposure to the intervention could have ended 
before 30 mins. During the interviews, the majority of players mentioned rarely thinking of this goal as it seemed 
unattainable and one mentioned not even realizing there was a larger goal. There are two potential explanations for 
this. First, asking users to choose the most difficult goal restricted participant agency and, thus, demotivated them. 
Secondly, diminished prominence of the goal on the user dashboard and the lack of language cues to motivate users 
toward that goal may have influenced their reactions. Instead, users tended to focus on “getting the likes and 
money.” 


== Options / Pause « 


Object Simulation. 

According to Fogg, object simulations have the key advantages of (1) fitting into the context of a person’s 
normal life, (2) being less dependent on imagination or suspension of disbelief, and (3) making clear the impact on 
normal life. 

Conceptually, participants stated that the game was representative of normal life on many different levels. 
Some mentioned the realism of the article headlines. Others noted the techniques used by disseminators. Two 
participants gameplay also represented their perspectives to bias and race in the U.S.: “Personally, I chose a white 
male each time I purchased a bot. Who are they going to listen to the most? This game represents America.” 

Although the game, as a concept, may have fit into the context of the participants’ normal lives, there were 
aspects of the game that participants felt were overwhelming dependent on imagination or suspension of belief. Four 
out of the seven participants specifically mentioned that it would have been helpful or entertaining to see “what the 
actual articles look like.” As one participant stated, “I never felt in any real way that I was making a website.” 
Another mentioned, “I expect to know what [my] ‘news’ site looks like currently.” 


RQ2: How does Fake It to Make It’s usability influence its perceived credibility as a persuasive media 
literacy tool? 

Although the majority of players expressed their enjoyment with the game, there were usability issues that 
frustrated some players. After observing 3.5 hours of participant gameplay and comparing interview data— 
specifically 24 coded responses related to usability issues—the researchers suggest that Fake It to Make It may have 
usability issues causing cognitive overload. For the purposes of this study, the researchers adhere to Mayer & 
Moreno description of cognitive overload—the processing demands evoked by the learning task exceeds the 
processing capacity of the cognitive system. (2010, p. 45) Although the researchers can only infer based on 
observation and thematic analysis, cognitive overload may potentially occur in this game due to (1) split-attention 
effect and (2) an overabundance of text and crowding of information. 

It is important to note that out of the 24 usability issue codes noted during interview analysis, 21 came from 
players who reported that they do not regularly play video games. Although the researchers cannot verify the 
information seeking skills of the participants, it is important to consider as those regular game players may perform 
more efficiently at information search and information retrieval. (Deza, 2016, p. 28) 


Split-Attention Effect 

Split-attention effect refers to the increase of cognitive load due to two or more sources of information that 
must be processed simultaneously in order to derive meaning. (Sweller, Van Merriénboer, & Paas, 1998) Although 
this effect was proposed in relation to instructional design principles, it applies to the experiences of the participants 
in this study; it occurred most frequently when participants were still learning the gameplay mechanics, specifically 
during the fourth goal of the game. Have an article that has earned at least $10. Players are required to complete 
four out of the five in-game steps for planting disinformation, which occurs down a series of drop-down panels on 
the left-hand side. While players do this, their disinformation submission information populates on the right. 
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During observation, it was noted that the “Review and Submit” drop-down panel was often clicked more 
than once; the player may have expected to be able to submit their article from the final drop-down panel. According 
to one participant, “I would lose track of where I was. Like, I would go through these steps on the left-hand side and 
then, oh, actually to plant, I had to go to the right-hand part of the screen. [And] that part was lower on the screen so 
I had to scroll down. I guess I was confused by the arrangement.” The remaining two participants with limited 
gaming experience noted this issues between the left and right sides of the screen as well: “It took me a little while 
to figure out you could join other groups. I was just stuck on the left-hand side.” “I never got all the way to the third 
button [to create a new profile, on the right]. So I don’t know why I didn’t notice that.” 


>» Hope Extra =e $336 [=| 


& Stel - . —— Ey — a ee 


«= Plant An Article 


Figure 6: The current design of the game forces players to process information simultaneously on two separate 
sections of the interface. 


Overabundance of Text 

Four out of the seven participants mentioned that the interface of the game was either “too busy” or 
contained too much text. The amount of text seems to be specifically related to the prompts or instructions between 
gameplay goals: “And then I realized I didn't read the instructions too much because... [it was] just too much text.” 
“T just make assumptions about what’ ll be there. So maybe I didn’t take the time to slow down and read it.” In 
addition to the amount of text mentioned, there were also general sentiments about navigating the screen and the 
number of options available. As one participant put it, “The screen was a little busy. It wasn’t super intuitive.” 


RQ3: Does Fake It to Make It’s current design promote procedural knowledge with claims on social 
media, i.e. verifying a post's sources and motivations? 

The researchers did not find any substantial changes between pre- and post-test literacy assessments 
following the rubrics provided by the Stanford Education Group. Additionally, correlations between demographic 
data and test results did not emerge. The pre- and post-test questions asked the participants to review a tweet’s 
credibility based on the source and the intended audience. In order to achieve mastery on the Stanford rubric, the 
respondents must identify the political motivations behind the source and evaluate the validity of the data—such as 
that the data presented in the tweets come from politically motivated polling firms. However, the game-play did not 
appear nuanced enough to support such refined views of disinformation and social media. The pre-test revealed five 
participants were already at the emerging level according to the rubric. Additionally, one participant was at the 
beginning level and one participant’s response was already at the mastery level. After taking the post-test, four 
participants remained at emerging and the beginning participant’s response did not change. The participant with a 
mastery score, however, dropped to emerging, and one emerging score dropped to beginning. Although, the 
researchers can provide conjectures as to why these scores dropped—fatigue after using the simulation, less readily 
apparent political motivations in the second tweet, feeling rushed for time—they cannot currently verify why these 
scores dropped. 
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Design Recommendations 


Recommendations for Increasing Persuasive Features 
In order to motivate users to continue to play Fake It to Make It and, thus, perhaps, influence player 
attitudes and behaviors, the researchers suggest the following design recommendations. 


Cause-and-Effect Simulation Recommendations 

To accelerate understanding of in-game 
actions, the researchers recommend that future 
versions make greater use of language cues and 
clearer indications of results after an article plant. For 
example, the player’s chosen avatar could explain 
why they succeeded or failed: “Nice work. You 
matched incendiary language the article with the 
rhetoric used by Orange Party supporters in this Snooze! People in this group 
group!” or “Snooze! This is a Purple Party social 
media group. Why are you posting articles about 


cats?” This may aid in guiding novice players through A 


didn't really seem to care about 
this article. Try something else. 


initial gameplay. 

Although participants mentioned the 
anticipation of seeing social media reactions, the 
effects of their in-game disinformation campaign 


could be emphasized. First, the reaction animations 

may become redundant. As one participant put it, “I Figure 7: An in-game reaction to player activities without an 
stopped paying attention to them and just clicked out of — explanation for why their planted article caused little response. 
the reaction space and quickly started making more 

articles. That was more fun because you could just see your money totals going up.” The player, in this instance, is 
seeing their in-game money growing, but they are not seeing the societal impact of their activities. According to 
Fogg, cause-and-effect simulation allow the user to vicariously experience the effects of their decisions. (2007, p. 
66) Future iterations of the game could increase the visual elements and show impact beyond the reaction 
comments. For instance, if an article goes viral, the player may accrue more ad revenues, but it comes at a real-life 
cost for others: an outburst of political violence, the tearing apart of a family, etc. Via text the game does show some 
consequences (such as an individual losing their job due to citing a player’s viral article), but the lack of physical 
cues—such as an animation—limits its impact. Witnessing the visceral effect of gameplay actions in this safe 
simulation may bolster the importance of verifying claims or news spread via social media. 


Environmental Simulation Recommendations 

Future iterations of this game should consider applying greater prominence to the goals of the game (e.g. 
purchasing a car) and situating it within context. Examples could include animated cut-scenes between tasks that 
include language that emphasize the importance of reaching a certain amount of money. These cut-scenes could use 
psychological social cues such as humor, personality, feelings, or empathy to try to motivate players to continue 
spreading disinformation (Fogg, 2003 p.91). For example, is this car needed to so the main character can drive their 
kids to school rather than them walking through a hostile neighborhood? Is it for the wild summer vacation on the 
coast of the Mediterranean? Additionally, the more visually attractive the cut-scenes are to the target audience, the 
more likely it is to be persuasive. (Fogg, 2003 p.94) These scenarios could be based on the player’s chosen avatar 
and, thus, increase the role-playing element of the game. 


Object Simulation Recommendations 

One way to increase the likelihood of transferring virtual behavior to the real world is to incorporate a 
virtual component into a real-world situation. (Fogg, 2003, p. 77) The game’s current design allows users to create 
or generate a site name, choose a logo, and then determine the amount of in-game money they would like to invest 
in making the site more credible. Adding the ability to see the front-end aspects—such as the websites the user 
creates and the profiles they purchase—may increase the realism of the game and thus advance Fake It to Make It as 
an object simulation. This could also situate it within the context of the player’s life as a social media consumer. 
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Recommendations for Increasing Usability 


Design Recommendations to Reduce Split-Attention Effect 
As the left-to-right navigation to plant articles may have increased extraneous cognitive load, the 
researchers suggest that future iterations of Fake It to Make It apply the gestalt principle of continuation. This should 
be applied to the drop-down panels that use four of the five functions for submitting an article. In fact, as one 
participant said, “Instead of having all this information on the right side, you could have just one smooth scrolling 
type screen.” This idea of one “smooth scrolling screen” is representative of player expectations of the gestalt 
principle of continuation; we follow and “flow with” lines (Interaction Design Foundation, n.d.). 
Design Recommendations to Reduce Overabundance of Text and Additional Navigation Issues 
For remaining extraneous cognitive load issues, the researchers recommend applying Mayer’s Cognitive 
Principles of Multimedia Learning (2017). Although the game does follow the cognitive principle of pre-training 
before each goal, providing the option to view information on key terms or features before having to work with 
them, the dual processing of large amounts of text and visuals is a hindrance. Following the modality principle, 
future iterations of the game could present information or instructions about a graphic verbally rather than as text so 
that learners can listen and refer to the graphical interface. (Mayer, 2017) 
Applying the signalling principle, also purported by Mayor, would benefit players. Signalling includes 
vocal cues and/or visual cues to aid the selection and organization of important information, especially for learners 
with low prior knowledge. (Mayer, 2017) 
Currently, the only signalling principles 


Goals . eee readily apparent is the prompt for users 
Create a site with a credibility rating of at least 30 to select the Help icon for more 

Add at least two articles to your site information about copying articles. 
aaealcrpomaiiatil ee 


even something as simple as marquee 
effect highlighting a certain feature of the 
interface. For example, this would help 


right-hand side of the screen may be 


Plant an article that triggers happiness 


clicked for even greater information. 
Figure 8: Currently, the text in Goals box seems static with no indication of interactive capabilities. 


Recommendations for Procedural Knowledge 


Increasing Manipulation during Article Writing 

After the player achieves certain goals in Fake It to Make It, they unlock the option to “write” articles to 
plant on social media. The simulation prompts users to write hyperbolic headlines and attach tags to make the story 
more believable, dramatic, or in-sync with a trending topic (e.g. veterans are being mistreated, a certain celebrity has 
died, etc.). The game allows players to add tags for increased believability (such as citing a verified resource), but it 
does not allow direct handling of, let alone show, concrete examples of manipulated information. The game should 
give the player actual statistical data about a topic and allow the player to alter it to fit a story they are about to 
share. With this additional game-play option, players may surpass the emerging level of the Stanford assessment and 
reach the mastery level. 


Manipulation for Profit vs Manipulation for a Cause 

The game situates the player as someone who profits from spreading targeted disinformation in the U.S. As 
the introduction screen reads, “You might not care about American politics, but you can still use its drama to profit.” 
Limiting the goal to be strictly monetary, however, may not aid individuals to be better equipped for verifying 
politically motivated information manipulation. In addition to the in-game goals that already exist, other game-play 
options could have players focus on a particular political purpose such as shifting the political leanings of a county 
during an election. This added level of detail has more real-world implications of how news is manipulated to push a 
specific narrative for a specific cause. This contextualized goal, if combined with the aforementioned object 
simulation recommendations, will provide more nuanced examples of disinformation, thus, potentially, increasing 
the procedural knowledge of the player. 
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Implications for Functional Triad and Human-Computer Interaction 


In 1996, BJ Fogg coined the term captology, and, in 2003, he expounded on the Functional Triad in his 
book Persuasive Technology: Using Computers to Change What We Think and Do. Since its inception, the Triad has 
received sharp criticism, ranging from ethical concerns to the incorporation of ‘categorical’ or definitial errors that 
do not stand up to theoretical scrutiny. (Atkinson, 2006, p. 173) First, although it is true that conscious as well as 
unconscious biases can seep into the product design, Warner undoubtedly makes her intentions clear at a time when 
media literacy is sorely needed. Second, the Functional Triad may have categorical errors—such as the term medium 
needing to be replaced with simulation—but these considerations do little to impact the use of the Triad as an 
examination tool in this study. In fact, its flexibility has led to its resiliency. The Triad allowed the researchers in 
this study to examine two forms of persuasion in the game: (1) Fake It to Make It’s motivational affordances to 
promote continued gameplay and (2) the impact of the game’s procedural rhetoric—its ability to “represent process 
with process,” which “only procedural systems like computer software” can accomplish. (Bogost, 2007, p.14) 


Conclusions 


Fake It to Make It?s Warner’s hope is that “by making players more aware of how and why fake news is 
written and distributed, that they will be more skeptical of what they encounter in the future.” (n.d.) A video game, 
with its ability to simulate an environment and show cause-and-effect relations, is an appropriate medium to 
consider for this goal. This study has shown that although Fake It to Make It provides an environment for some 
exploration and entertainment, there is room for improvement as a cause-and-effect, environmental, and object 
simulator. The researchers suggest providing quicker, more poignant feedback, greater emphasis on in-game goals, 
and including more realistic elements. Additionally, by drawing upon the cognitive principles of multimedia 
learning and reducing split-attention effect with gestalt guidelines, the usability—and thus credibility—of the game 
will increase. Lastly, although Fake It to Make It provides a glimpse at disinformation as a means for wealth, it does 
not expound upon the motivations and minute techniques used by politically motivated by actors. By doing so, 
procedural knowledge with social media claims may increase as well. With these recommendations, Fake It to Make 
It may become a useful—and fun—persuasive game. 
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Abstract 


This paper presents a successful 3-year (2016-2018) summer program, a grant project funded by the National 
Science Foundation (NSF) to provide students with international research experiences. The success encompasses 
three important components — program design, deep level of learning engagement and preparing students for future 
career in the globalized job market. A grant evaluation has been integrated into the entire program to improve the 
quality of the program. This study indicates students not only get know Chinese culture but more important they 
understand their own culture better through the eyes of different culture. UH cross-cultural learning program 
effectively prepared the students for 21st century’s globalization. 

(For detailed summer program information, visit http://ires.nsm.uh.edu/) 


Keywords: cross-cultural learning, globalization 
The Summer Program 


The University of Houston (UH) has been awarded an IRES (International Research Experiences for 
Students) grant from the National Science Foundation (NSF) to create an eight-week summer program to fund 
students to study and research for three years (2016-2018) in China. The mission of the grant is to prepare U.S. 
college students to be diverse and globally-engaged research leaders equipped with world-class research skills to 
strengthen economic competitiveness in this fast technology-emerging and globalized world. The traditional and 
enclosed classroom is no longer effective and powerful enough to educate our students with ready-to-use knowledge 
to solve real world problems when they step out of school. Globalization needs students to acquire new knowledge 
and skills that can help them survive and compete. They need to develop new world ability in addition to research 
skills. Cross-cultural understanding is one of the most challenging issues in cross-cultural research activity. The UH 
Summer Program is one of the best solutions to this challenge by providing students with opportunities to 
understand the cultural differences with their own personal experiences through living and studying in a foreign 
environment, fully merging themselves in the new culture and surrounded by local people while conducting on-site 
research. The program “provides students with a global context in which students can improve their cultural 
awareness and gain social competency to accommodate the changes in the global community” (Liu, 2007, P.36). 

The summer program is a collaborative work between UH and the China University of Geosciences (CUG) 
at Wuhan. UH and CUG developed diverse activities for students to facilitate and enrich their cultural learning 
experiences. UH organized three orientations before students leave for the international trip. For majority of 
students, it was their first time to travel abroad. The orientation from the UH Study Abroad Office helped them with 
travel procedure, policies and rules. The second cultural orientation introduced students about Chinese culture, 
holidays, food, social system, as well as the unique communication styles, and more importantly how to respect 
cultural differences and hold proper attitude to address the conflicts on cultural values. The third orientation is to 
prepare students academically on how to conduct field landslide investigations using state-of-the-art GPS and 
LiDAR techniques. CUG created twenty-four Chinese language courses, cultural activities and events in addition to 
research activities to help students better understand Chinese culture 
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The 3-year summer program had successfully accomplished in summer 2018. Throughout the 3-year 
implementation, a grant evaluation has been integrated to collect data from each student activity to overall 
effectiveness of the program. The results from the data analysis have been used to improve the quality of 
implementation. Three important aspects were summarized from this successful implementation, the designing of 
the program, student deep-level engagement in learning experiences, and align academic curriculum with societal 
needs to prepare students to be competitive and responsible researchers in their future career. 


I. Design for Success 
Pre-departure Orientations in US 


There were altogether twenty-two students participated in the summer program, eight students in both 2016 
and 2017, and seven students in 2018 Female students outnumber male students by 3. The ethnic background of 
students is very diverse, twelve Caucasians (55%), four Latino or Hispanic (18%), four Asian or Asian American 
(18%), one African American (5%) and one Native Hawaiian (5%). Their age range is from 20 to 36. Four students 
had never travelled abroad prior to the summer program. Only one student had study abroad experience before. In 
order to prepare students for successful cross cultural learning experiences, the summer program very carefully 
designed a series of activities. Three pre-departure orientations were organized to help students smooth into the new 
cultural environment with readiness for their living and learning in a foreign country. The UH Learning Abroad 
Office (https://www.uh.edu/learningabroad/) delivered a presentation on passport and visa etiquette, which was 
especially helpful for the students who had never traveled outside the country, and instructed students to keep paper 
and digital copies in various places in case any unexpected situation occurred. Students were also provided with 
detailed travel tips regarding immigration, carry-ons and luggage, medication abroad, research weather and culture, 
Wi-Fi and technology. The office also helped students understand culture shock, and coping strategies on how to 
deal with discomfort, confusion and frustration while living abroad. The office advised students to be engaged in 
and explore the new culture, make friends with local people, maintain an appropriate level of contact with friends 
and family back at home as well as keep an open mind and avoid judgement. Safety abroad, alcohol and drug law 
were also included in the presentation. For most students, they had never lived or studied in China, they had little or 
very superficial knowledge about the country and the places they would go. The presentation helped students to be 
aware of differences, rules and regulations in the local culture. For the past three years, students had improved rating 
for the presentation. In 2018, seventy-one percent of students thought it was very helpful, and 29% thought it was 
moderately helpful. Table 1 shows the comparison ratings in the past three orientations from the UH Study Abroad 
Office. 


Table 1. The Comparison Ratings in the Past Three Orientations from the UH Study Abroad Office 


Year Very Useful Moderately useful Slightly useful 

N % N % N % 
2016* 2 29% 1 14% 4 57% 
2017* 5 M1% 1 14% 1 14% 
2018 5 1% 2 29% 0 0% 


*Seven among eight students finished the survey. 


The second pre-departure orientation is specific about Chinese culture, major holidays both traditional and 
governmental, Chinese cuisine in different areas, unique characteristics of Chinese culture. The orientation is not 
just providing apparent cultural information, but more importantly it intends to educate students the nature of 
culture, which is not inherited but learned. Culture forms values, creates attitude, and influence behaviors. Students 
need to realize that culture is dynamic and the traditional stereotyping value system is not applicable in today’s 
world. Since China adopted the open policy in mid-80, more and more young people have been exposed to the 
western world and they have been embracing western value in their life. This creates an optimal environment for 
foreigners to live and study in China since young people are more open and communicative to people from outside. 
However, the old traditional value is still prominent enough to present challenges to foreigners to live in the mixed 
culture, such as the priorities of cultural values that are different between US and China. People tend to 
communicate in directness and openness in US, while Chinese are more indirect and their communication is often 
embedded at many socio-cultural levels. Mutual respect is seen as the norm in US, but in China “Equality of 
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treatment is not assumed nor is it necessarily regarded as desirable” (Nisbett, 2004, p. 49). “The traditional Chinese 
moral system clearly spelled out the relationship between officials and ordinary people, parents and children, older 
generation and younger generation (Liu, 2007, P. 39). The cultural presentation helped students understand cultural 
differences so that they can communicate properly in the new environment. Consciously understanding the blocks 
to cultural communication, such as ethnocentrism, discrimination, stereotyping, culture blindness and cultural 
imposition are extremely important in the globalized society. Students were offered the skills to overcome cultural 
differences and encouraged to respect differences and to keep their personal beliefs personal without judging others 
while living and working together with local people or even within their own group. In the situations where the 
conflicts occur, students were provided with skills to resolve the conflicts through understanding others in their 
cultural perspectives. Students reported positive feedback on the cultural presentation as shown in Table 2. 


Table 2. The comparison ratings in the past three presentations on Chinese culture 


ge Very Useful Moderately useful Slightly useful 

N % N % N % 
2016 1 14% 5 11% 1 14% 
2017 5 l1% 2 29% 0 0% 
2018 5 11% 1 14% 1 14% 


GPS and LiDAR technology training is the third orientation activity. This training session prepared 
students with the knowledge necessary to conduct successful field research on landslide monitoring in China. This 
session covers the topics on GPS and LIDAR technique development and their applications to studies of earthquake, 
landslide, subsidence, faulting, coastal erosion, sea-level change and structural monitoring. After the first summer 
program, the instructor, also the PI (Principle Investigator) of the grant made some changes and added more hand-on 
practice and reading materials to address student needs and improve their level of understanding and ability to 
conduct field research. This training session shows the greatest improvement in the past 3 summers. In 2018, 100% 
students thought the training session is very helpful as shown in Table 3. 


Table 3. The Comparison Ratings in the Past Three GPS and LiDAR Training Sessions 


vax Very Useful Moderately useful Slightly useful 

N % N % N % 
2016 1 14% 1 14% 5 11% 
2017 5 M1% 2 29% 0 0% 
2018 7 100% 0 0% 0 0% 


Language Classes and Cultural Activities in China 


In order to facilitate student living and learning and enhance their research experiences in China, the China 
University of Geosciences (CUG) at Wuhan created eight language and cultural classes to improve students’ 
language skills and to update students with today’s China. The instructors engaged students learning through direct 
interaction, playing games and using different strategies to help student practices. The class ratings were improved 
through the years. In 2018, 100% of students thought that the Chinese phonetics, lexis, and daily communication 
dialogue classes were very helpful, 86% thought the Chinese language class was very helpful. China Today and 
Chinese Culture classes were also reported very help with 71% ratings. Students stated the following most helpful 
and fun activities in the language classes in summer 2018 program. 

Speaking with people outside of the classroom utilizing material learned in class. 

The language classes, the pronounce classes, the China today class. 

Learning ping yin to start grasping on local language and being with locals (volunteers and/or teachers). 

The best way to learn the language was simply to be immersed in the language. 

Practicing what you learned in class with actual Chinese speakers and calligraphy class. 

I learned the most during the "directions" class. We had to work in pairs and give directions to our 

blindfolded partner to find everyone around the room. All the paired language activities were very useful. 

Besides that, I learned the most just hanging out with the volunteers outside of class. 
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Learning ping yin, practicing and repeating, reviewing materials we learned in classes, playing games using 
material we learned in class, smart boards that teachers used so they can emphasize materials and write 
over presentations when needed, and spending time with locals/volunteers 

The language classes were great, especially when the teachers would have interactive games to help us 
learn the language. 

Practicing Chinese with the teachers was the most helpful because I got to work on my pronunciation. 
When teachers had games for us to reinforce what they were teaching at the moment. 


CUG also arranged sixteen cultural activities to help students better understand Chinese culture. The 
activities range from visiting museums (Hubei Museum, Museum of 1911 Revolution, Museum of CUG, and the 
State Geological Museum in Beijing), sightseeing (the Great Wall and Tiananmen Square in Beijing, the East Lake, 
Moshan Mountain, Yangtze River Cruise, Hankou Bund), to Chinese folk activities (Chinese zodiac paper-cutting, 
Chinese painting and calligraphy, Tai Ji/Kung Fu, Chinese knot, jewelry appraisal and appreciation, and watching 
Chinese movie). In three summer programs, students rated Tai Ji/Kung Fu, visiting the Great Wall and Chinese 
calligraphy as the extremely helpful activities to understand Chinese culture. Students in 2018 group also showed a 
great interest in visiting the museums. Table 4 shows the ratings on the top three activities in the past three summer 
programs. 


Table 4. The Comparison Ratings for Top Three Chinese Cultural Activities 


Ne he Extremely helpful Very helpful Moderately helpful Slightly helpful 
Cultural Activities 
2016 | 2017 | 2018 | 2016 | 2017 | 2018 | 2016 | 2017 | 2018 | 2016 | 2017 | 2018 
Tai Ji / Kung Fu 57% 37% 29% 14%} 25%) 57%) 29%) 12.5% 14% 0% 25% 0% 
Chinese Calligraphy 43% 50% 29% 43% 38%) 57%) 14%} 12.5% 0% 0% 0% 14% 
Visiting the Great Wall 29% 63% 57% 14% 25% 14%) 43%) 12.5% 14% 14% 0% 14% 


Integration of Grant Evaluation 


The summer program integrated a research component throughout the entire grant implementation. The 
research study is instrumental to the success of the program. The analyzed results of the collected data were used to 
improve the overall quality of the program. A pre-survey and a post-survey were designed to collect data. For the 
first two years, students were surveyed before and after their trip and the data analysis were entirely based on 
students’ perceptions and feedback from their living and learning experiences in China. For the last trip, the grant 
evaluator participated in some research activities and interacted with students and observed their learning, which 
provided very valuable data for the research study. Students were fully engaged in the exploration and investigation 
process and showed a great interest in new knowledge acquisition. The evaluator witnessed the true impact of the 
program on student learning and research activities. 

A pre-survey was conducted before students leave for the trip. The survey has two sections. The first 
section is to collect student demographic information and to find out student’s foreign travel experiences, their 
preparation for the trip, as well as their expectation from the summer program. The collected data on student 
background, their foreign travel experiences and their expectation of the program greatly helped the PI better 
understood the students and appropriately address their needs during the program implementation. Their 
expectation from the trip was two-fold. Academically, students wanted to study geology to develop professional 
expertise, to improve the knowledge of GPS and LIDAR, to use technology through hands-on research activities and 
to explore geological changes in earth structure in the real world. Culturally, they wanted to learn more about 
Chinese cultures, language, and to expand their world view. The second part of the pre-survey is to assess student’s 
perceptions and attitude toward foreign cultures. There are twenty-eight questions in section two, asking students’ 
comfort level of interacting with people from different cultures, their attitudes towards accepting the values of 
different cultures. The data on the second part in three summer programs indicate that all students expressed their 
interest in learning about many cultures that have existed in the world, feel comfortable in making friends from all 
ethnic groups and respected their friends’ opinion of their own culture even if they disagreed with them. Ninety-five 
percent of them strongly agreed or agreed that it is important to respect differences in individuals from diverse 
cultural backgrounds, if they need more information about a different culture, they would feel comfortable asking 
people from that culture and people from different cultural backgrounds are willing to help each other. More 
importantly, they all strongly agree or agree that people from different cultural background can be trusted. However, 
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in 2016 group, twenty-two percent of students did not think that all cultural values are equally important. Based on 
this result, the cultural orientation was changed to focus more on cultural and racial equality education and stress on 
mutual respect of different cultures and their value systems. 

After students came back from China, a post-survey was conducted to gather the data related to their 
attitude toward cultural adaptation and acceptation through their studying and living experiences in China. The 
survey has six sections. The first section is to evaluate the three pre-departure training activities before students’ trip. 
The second section is collecting students’ feedback on the courses about the helpfulness toward learning Chinese 
language and understanding of Chinese culture. At the end of this section, there are four open-ended questions, 
asking students about their most and least enjoyable activities, most helpful activities and best methods to learn 
Chinese language. The third section is the ratings on the Chinese folk activities regarding their helpfulness toward 
understanding Chinese culture. The forth section is about students feedback on their travel experiences related to the 
benefit of the trip, understanding of Chinese culture, communication with local people, cultural shock, adaption to 
new culture, comfort level in making friends in different culture, increased interest in learning about a new culture, 
perceptional changes towards Chinese culture and their own cultures based on their study and life experiences. 
Students have an opportunity to provide extra information on the most challenging aspects for them to study and live 
in China. The fifth section of the survey is to gather students’ feedback on regional geology and landslide study, 
rating the educational and supporting facilities at CUG field camp in Zigui, field activities in Zigui and Huangtupo 
landslide site. Students also provided suggestions on the improvement of the field investigation at the Huangtupo 
landslide site and their difficulties of conducting field geological study in China for foreigners. The last section is 
students’ feedback on the overall quality of the program regarding meeting the expectations, organization, 
satisfaction, integration into academic study, and impact for the future career. Each year’s data have been used to 
guide the improvements for the following year. Table 5 shows students’ feedback on their travel experiences for all 
three summer programs. 


Table 5. The Comparison Ratings of Trip Experiences in Three Summer Programs 


Strongly Agree Agree Disagree Strongly Disagree 


Trip Experience 
2016 | 2017 | 2018 | 2016 | 2017 | 2018 | 2016 | 2017 | 2018 | 2016 | 2017 | 2018 


You were able to understand 
the Chinese culture with what 0%} = =S7%) = 43%! ST7%} 43%) =—S7%) 29%) = 0%) OMS 14%) O% 0% 
lyou had learned 

You were able to communicate 
lwith local people with the 0%} 29%} 29% 29%] 71%! 71%! 57%) O% O%} 14% 0% 0% 
(Chinese you had learned 
You were surprised by some 
Chinese cultures that were new 14%) = 29%) = S7T%}- 43%) 71%) 43%) = 14%) = O% OM 14%) = OM 0% 
to you. 

You were able to adapt to the 
local culture easily 

This trip increased your 
comfort level in making friends 14%) = 43% = S7T%}-S7%) 43%) 29%) = 14%) = OO% OOM 14%) — ON 0% 
from different culture. 

This trip increased your interest 
in learning about a different O%} = 85%) = 85%! 85%} =—14% «14% = 14%] = 0%) OM 0%) 0% 0% 
culture. 

This trip positively changed 
lyour perceptions about Chinese 29%| S7% 71%! = 43%) 43%) 29% 14%! O% O% 14%) 0% 0% 
culture. 

This trip helped you better 
lunderstand your own culture. 
Your previous foreign travel 
lexperiences benefit you on this 29%} S7%| = ST%! 43%) 43%) 29%) 29%! 0% 14% 0%} 0% 0% 
trip. 


29%) 43%) 43%) 43%) 57% 43%) 29%! 0% 14% 0%) 0% 0% 


14%) = 43%) = 43%} 43%) 57%) = S7T% = 43%) OMS O% 0%) 0% 0% 


The dataset indicates that the summer programs play a very important role in promoting cross-cultural 
communication and understanding. In 2017 and 2018, 100% of students either strongly agree or agree that they were 
able to understand the Chinese culture and able to communicate with local people with what they had learned, the 
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trip increased their comfort level in making friends from different culture and increased their interest in learning 
about a different culture, the trip positively change their perceptions about Chinese culture, and more importantly 
their Chinese experiences helped them better understand their own culture. These trip experiences equip them with 
new skills and knowledge to be more competitive and ready to work with people from different cultural background 
in this globalized work environment. The research study provided very valuable information on the impact of the 
trip in student future life and career. 


Il. Deep-level Learning Engagement 


Engagement has been positively correlated with student achievement (Salamonson, Yenna, & Everett, 
2009), a positive student experience and increased student satisfaction, student resilience and mental well-being and 
more active alumni and life-long learners (Markwell, 2007). Harper and Quaye (2009) define student engagement as 
“participation in educationally effective practices, both inside and outside the classroom, which leads to a range of 
measurable outcomes” (p. 2). Akey (2006) asserts engagement is “the level of participation and intrinsic interest that 
a student shows” (p. 6) and which involves behaviors, attitudes and affect. The learning activities in the summer 
program provided students with an excellent opportunity to fully engage them in the process of research and practice 
in the real world. Students were very actively involved in exploring and investigating the evolving changes on the 
earth, which they had no chance to see in a traditional classroom. 

The major research activities were carried out at the GUG campuses of ZiGui and Badong. Zigui is a 
county of western Hubei province. It is situated on the high southern shore and encompasses the easternmost portion 
of the Yangtze River Gorges, including the Xiling Gorge and it is a few kilometers west of the Three Gorges Dam, a 
hydroelectric gravity dam built between 1994 of starting to 2012 of full operation. The Dam has brought both 
benefit and potential hazards to the region. It generated 97.6 billion kilowatt-hours of electricity in 2017 (Li, 2018). 
The goal of the project is to “fuel China’s economic development using clean and cheap energy and to prevent the 
emission of carbon dioxide” (Li, 2018) from coal generated electricity. The dam project has negative environmental 
impact in the region, such as erosion, sedimentation, earthquakes and landsides. The three gorges are thought to 
have been formed by river incision of massive limestone mountains of Lower Palaeozoic—Mesozoic age in response 
to episodic tectonic uplift during the Quaternary (Li et al., 2001). Elevation ranges from 800 m to 2000 m above the 
sea level, and the terrain comprises a succession of limestone ridges and gorges, with inter-gorge valleys where 
interbedded mudstones, shales, and thinly bedded limestones predominate. Since the objectives of the summer 
program are to study the geological structure, evolving changes and landsides, this region is a perfect match for the 
intended research purpose. 

The research activities at Zigui were led by the professor Hanwen Zhou from CUG at Wuhan. Students 
followed him to different sites to explore and investigate the geographic characterizes in the mountains. The 
professor explained in detail the earth structure, unique stone formation, such as the Liantuo Formation, sandstone, 
the moderately thick-layered and moderately coarse feldspar quartz sandstone, cross bedding of sandstone, the 
endurance and longevity of the stones. Students asked questions related to the knowledge they learned in school and 
clarified the confusion with real samples in front of them. They dig samples and used magnifier to study the ultra- 
fine details of the stone. Photos | through 4 show the engagement and interaction between the students and the 
professor. 


Photos 1-4. Conducting Research Activities at Zigui Location 
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Students learned new knowledge from their own experiences and they demonstrated a great interest and 
enthusiasm on every expedition. They were very excited through their participation in hands-on activities as 
observed by the grant evaluator, who was on the site with students. Some students even carried heavy samples in 
their bag with them and they planned to bring them back home. The second location of research activities is at 
CUG’s campus at Badong town to study the Huantupo landslide. The following Table 6 shows students’ feedback 
on research activities at these two locations. 


Table 6. Students’ Rating of Zigui Research Activities 


Like it very much Like it Somewhat like it Did not like it 
2016 | 2017 | 2018 | 2016 | 2017 | 2018 | 2016 | 2017 | 2018 | 2016 | 2017 | 2018 


Activity Locations 


The educational and 
supporting facility at B%) TI% 86% 43%) 29% 0%) 29% 0% 14%, 14% 0% 0% 
Zigui 

The field activities at 
Zigui 

The filed activities at 
the Huangtupo landslide} 43%) 71%) 57%! 14%) 14%) 28%} 28%) 28% 14% 14% 0% 0% 
site 


43%] 86% 100% 57%} 14% 0%) 0% 0% 0% 0% 0%) 0% 


The dataset shows that in summer 2018, 100% students liked the Zigui activities very much. In both 2016 
and 2017, 100% of students either liked it very much or liked it. Besides ratings at Badong location, students 
provided suggestions to improve the activities. The hands-on activities truly engaged students in learning new 
geological knowledge. They were very excited when they found out something unique, such as insect fossil. The on- 
site research activities prepare students professionally with the ready-to-use skills to work in a real world 
environment and at the same time they can improve their cross-cultural understanding, language and communication 
skills. 


Ii. Preparing Students for Future Success 


The Partnership for 21st Century Skills believes that making the connection between learning and the real 
world is imperative for student success. According to the Partnership, “the education system faces irrelevance unless 
we bridge the gap between how students live and how they learn.” The Partnership defines literacy to mean not just 
reading, writing, and computing skills, but “knowing how to use knowledge and skills in the context of modern life” 
(21STcentruryskills.org). UH summer programs are the best practice in bridging student learning in classroom with 
real world living and working environment while gaining skills for cross-cultural understanding and communication. 
The programs are an excellent alignment of student learning with societal needs. Traditional classroom research 
projects detach students from reality. The knowledge and observation they gained are mostly from textbooks and 
samples that already prepared for them. In the summer programs, students used tools to search for the samples on 
their own and they were developing the skills through examination and analysis. They also exchanged their 
experience with each other and solicited guidance from professors to develop team-work ability to collaboratively 
making progress together. This process of learning prepares student’s research skills as well as improves their 
communication and team work ability and those soft skill sets are highly sought out by today’s employers. In the 
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summer program, students not only conducted hands-on research activities, but also had opportunities to 
communicate with local people, with the local students who worked as volunteers to help them with their language 
learning and other needs. Students had a lot of fun in their interaction with people in China. In summer 2018, the 
CUG professor took students to a local farmer’s residence in the Lianziya landslide area, and students had a chance 
to learn more about people’s daily life there. The following two photos show students were relaxing and visiting the 
farmer’s house and taking a group photo. The couple was very excited about the visit, and they cooked some food 
for students, eggs directly from the chicken coop, and peaches from the trees, cucumber and tomatoes from vines. 
For the very first time, students ate boiled peaches. 


et 4 Nagin 


(1 min 


In the globalized world, obtaining people skills is as important as technology skills. Being able to 
communicate effectively with people from different culture is no longer a plus skill but an essential skill to be 
competitive in the job market. The summer programs offer an important avenue for students to achieve multiple 
goals through learning new geological knowledge, improving research skill and developing communication ability. 
This real world approach of learning greatly enhance the quality of education model by students walking out of the 
ivory towel and bringing in real life experiences into the classroom. The following Table 7 shows the students 
response to the positive impact of the program on their future career. 


Table 7. Students’ Responses on “Will this program have a positive impact on your future career?” 


Year Strongly agree Agree Disagree Strongly disagree 
2016 3 43% 2 28% 1 14% 1 14% 
2017 5 71% 2 28% 0 0% 0 0% 
2018 4 57% 3 43% 0 0% 0 0% 


With program improvements, both programs in 2017 and 2018, 100% of students either strongly agree or 
agree that the program will have positive impact on their future career. When students were asked if they would 
recommend this program to other students and why, they stated the following. 


Yes, I would definitely recommend this program to other students. It represents a great opportunity to 
learn Chinese language and infrastructure technologies. 

Definitely yes! I expected to just do Geology field work like usual but it ended up feeling like a full 
study abroad experience. I made friends and immersed myself in the culture. I would especially 
recommend this to someone who perhaps doesn't have the time to spend a whole semester abroad. 6 
weeks was enough time to at least start really getting immersed in the culture. 

Yes, it is a great opportunity to engage in geoscience research and learn methods from another country 
that could impact positively on our country. Also, it is a great opportunity to study abroad and learn 
from a different culture other than your own. 

Yes. I enjoyed the experience and would recommend it to others. I was able to learn Chinese culture 
and language while studying geology and enjoying my time. 
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Conclusion 


Globalization imposes new challenges to the traditional educational system. It is not enough just to educate 
all-A students. Brand et al. states that students must not only acquire knowledge, they must develop the ability to use 
knowledge, skills, and dispositions to evaluate and efficiently solve novel problems if they are to become what 
society need (2000). Schools need to train students to be responsible citizens and to be competitive in both academia 
and reality with ready-to-use knowledge and skills to tackle real-life problems while communicating effectively with 
people from different cultural background. UH summer programs created an optimum educational environment that 
maximized learning resources and provided students with a meaningful learning. Students had an opportunity to 
collaborate, network and exchange ideas with each other and cross-culturally, to conduct geological research 
through exploration and investigation and apply knowledge in a novel setting. A successful education is all about 
creative learning and live practice of knowledge, linking academic engagement with social engagement so that 
students can make an operational connection between school and reality. Knowledge is constructed and influenced 
by social interaction and students are more likely to retain and transfer knowledge when given opportunities to apply 
what they are learning to real world issues (Melaville, Berg & Blank, 2006). An effective cross-cultural 
communication is crucial in the globalized job market. In the summer programs, students learned to understand and 
respect a new culture through full engagement and interaction with local people and improve their personal 
communication skills through ethical, social and professional understanding. In this process, they not only get know 
Chinese culture but more important they understand their own culture better through the eyes of different culture. 
UH cross-cultural learning program effectively prepared the students for 21st century’s globalization. 
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Abstract 


This paper presents a research study to incorporate Artificial Intelligence (AD) tools into Civil Engineering 
(CE) and Electrical/Computer Engineering (ECE) undergraduate curriculum at Texas Southern University. The 
infusion of innovative ECE specialized AI tools into traditional CE problem-solving routines by problem-based 
learning (PBL) approach was designed to enhance engineering students! problem-solving and critical thinking skills, 
expose them to new technology, prepare them for diverse and multidisciplinary workforce requirement. In the 
process of the implementation, multiple assessments have been used to measure student success for both academic 
content knowledge and broader competencies beyond classroom. The multi-dimensional assessments included direct 
assessments to evaluate student learning activities, an indirect assessment to collect student feedback as well as the 
national Course-based Undergraduate Research Experience survey to find out student learning performance as 
compared to the national level in STEM field. The collected data have been used strategically to make project 
expansion, enhance teaching quality and improve student learning both in and beyond classroom. 


I. Introduction 


Texas Southern University (TSU) is located at Houston, the fourth largest city in US. TSU is ranked as the 
second nation’s largest HBCU (Historically Black College and University) by enrollment with 12 colleges and 
schools and offering more than 100 undergraduate and graduate programs and concentrations. College of Science, 
Engineering & Technology has the largest enrollment every year and educates students with knowledge and skills 
for the job market. The National Science Foundation created a grant program HBCU-UP (Historically Black 
Colleges and Universities Undergraduate Program) to enhance the quality of science, technology, engineering and 
mathematics (STEM) instructional and outreach programs at HBCUs as a means to broaden participation in the 
Nation’s STEM workforce. This research project has been funded for four years from 2016-2019 by the National 
Sciences Federation to infuse Artificial Intelligence (AI) tools into Civil Engineering (CE) and Electrical/Computer 
Engineering (ECE) undergraduate curriculum at Texas Southern University. The infusion of innovative ECE 
specialized AI tools into traditional CE problem-solving routines by problem-based learning (PBL) approach was 
designed to enhance engineering students' problem-solving and critical thinking skills, expose them to new 
technology, prepare them for diverse and multidisciplinary workforce requirement. It also helps bridge current 
curricula gap in the Department of Engineering at TSU. The objectives of the project are: 1) to develop an intelligent 
knowledge database to document, compare, and analyze cutting-edge AI applications in civil engineering field, 
which can be used as the platform and educational media for curricula development and implementation for PBL 
approaches in classroom; 2) to add one new interdisciplinary course to the Department’s curricula “AI Tools for 
Engineering Problem Solving” for all senior engineering major students; 3) to enrich current curricula by integrating 
innovative AI application case studies into more than fourteen existing courses being offered in the Department, 4) 
to foster interdisciplinary academic setting by hosting server-based intelligent database in the College of Science, 
Engineering and Technology and to provide web- and classroom-based workshops and tutorials to all interested 
students and faculty; and 5) to support undergraduate students’ early involvement in research. 
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One of the main goals of the project is to promote learning beyond classroom by fostering interdisciplinary 
academic settings and to support undergraduate students’ early involvement in research. In the process of the 
implementation, four undergraduate courses in two academic programs have been selected in Spring of 2018 as pilot 
studies: Civil Engineering (course number identified with prefix CIVE) and Electrical/Computer Engineering 
(course number identified with prefix CMET): 


Freshmen level core course: CIVE141 Civil Engineering Materials. The course is being offered twice per year. 
Sophomore level core course: CIVE 224 Geotechnical Engineering. The course is usually being offered once 
per year in Spring. 

Senior level core course: CIVE 339 Reinforced Concrete Design. This class was originally offered once per 
year in the Fall semester, but due to increased interests and enrollment from junior senior students it was offered 
in Spring of 2018 as an addition. 

Junior/Senior level elective course: CMET 438 Artificial Intelligence. This course is being offered once per 
year in the Spring. 


Student learning is affected by different factors. It is very essential to collect precise and wide-ranging 
assessment data from difference sources in order to make effective improvements in course design and instructional 
quality to enhance student learning. In order to expand assessment strategies beyond classroom, which include 
competencies such as the student outcomes specified in Accreditation Board for Engineering and Technology 
(ABET) criterion 3a-3k"”: 


An ability to function on multidisciplinary teams 

An ability to identify, formulate, and solve engineering problems 

An ability to communicate effectively 

The broad education necessary to understand the impact of engineering solutions in a global, economic, 
environmental, and societal context 

A recognition of the need for, and an ability to engage in life-long learning 

A knowledge of contemporary issues 

An ability to use the techniques, skills, and modern engineering tools necessary for engineering practice. 

These competencies are not “knowledge” that can be captured in traditional tests. A series of assessment 
activities were carefully designed aiming to effectively measure student success for both academic content 
knowledge and broader competencies beyond classroom. 

A good assessment practice shifts the focus from blaming students for not achieving expected learning 
outcomes to finding out true reasons behind student learning issues. The grant project implemented multi- 
dimensional assessments. Student learning outcomes are evaluated through direct assessments in course 
assignments, projects, group activities, quizzes and exams, and student grades have been used to analyze student 
class performance at the same time to provide meaningful information for course improvements. In addition to the 
direct assessments in class, two indirect assessments have been conducted to collect data on student learning in and 
beyond classroom from their perspective. Both indirect assessments used survey method. The Research on the 
Integrated Science and Engineering Curriculum (RISEC) survey was developed in-house. This survey is specifically 
targeting at student learning and using the AI tools in their hands-on projects and research activities to find out 1) 
student engagement and interaction in the process of learning, 2) student research and problem solving skills, 3) 
student feedback on how the artificial intelligence infusion project contributed to their course contents learning, and 
4) student open feedback on overall effectiveness of instructional delivery and project implementation. The second 
survey is the Classroom Undergraduate Research Experience (CURE) survey, which has been used nationwide to 
measure student experiences in "research-like" or other science courses. In this project, CURE data is used as a 
bench mark to compare TSU students with nationwide students in STEM field in science attitude, learning 
experiences, learning gains, learning benefits and overall evaluation of the course. The survey results give the 
instructor an opportunity to find out the course teaching effectiveness, and more importantly to compare TSU 
students with national counterparts in academic performance. The additional value of these surveys is to find out the 
impact of AI infusion on student learning and their future career. The indirect assessment data contribute to the 
project’s success in several aspects. Firstly, the data have been used to make instructional improvement, secondly, 
student feedback provided very important information for AI project expansion, thirdly the data help the instructor 
to compare the performance of TSU students with national peer in STEM field, and lastly, the data provide evidence 
of the impact of AI tools on student learning beyond traditional classroom knowledge. 


191 


II. AI Infusion to Promote Learning beyond Classroom 


The overarching technical justification for integrating state-of-the-art AI tools with state-of-the-practice 
engineering methods is the rapid advancement in AI technologies, which has made it a powerful and ubiquitous 
solution for many complex engineering problems. The Engineering Department of TSU has two majors: CE and 
ECE. Currently CE curricula focus exclusively on conventional mathematics, physics, and/or engineering methods 
for core engineering courses, and these classical curricula are highly specialized in solving different types of 
problems such as optimization, simulation, prediction, modeling, diagnosis, design to name a few. On the other 
hand, ECE major students learn AI theories and algorithms in depth but lack real case engineering applications in 
their curriculum to fully appreciate the knowledge they are learning. As pointed out by famous educator Malcolm 
Knowles, adults learn best when they understand why something is important to know. Hence four courses were 
selected as pilot studies to promote students learning outcomes. Figure | depicts the flowchart of the pilot studies. 
Three representative CE core courses at different levels (freshmen CIVE 141, sophomore CIVE 224 and 
junior/senior CIVE 339) and one junior/senior level ECE elective course (CMET 438) were selected for the 
implementation. 


AI Tools to be used 


-| | Evolutionary Fuzzy Logic o 
-| | Computation 4 


How can AI tools help me in 


solving real world problem? 


Knowledge- || Artificial 
Based ra Neural 
Systems }| Networks 


Why it is practical for me to 
learn AI tools? 


Common Constrains for both CE and ECE 


Missing Information 
Noisy or poorly understood relationshi 


y 


Jonnections 


Figure 1. AJ Infused Curriculum Implementation Flowchart 


For the three CE courses, AI infused curricula were introduced in three phases: firstly, a self-developed 
intelligent database was used to help students learn the basics of commonly used engineering thinking strategies, 
their state-of-the-art counterpart of AI assisted strategies and specific domain knowledge relates to each individual 
course; secondly, one or two carefully selected case studies were presented to each class with group discussions, 
both in class and through blackboard discussion board to guide students learn how to explore alternative candidates 
and search for solutions; lastly, by the end of the semester a hands-on term project was then assigned to each class. 
According to National Survey of Student Engagement (NSSE), a leading authority dedicated to improving 
undergraduate education, such curriculum has the pedagogical merit of engaging undergraduates in more in-depth 
thought which will provide undergraduate students greater educational and personal gains!, 

For the ECE course, adoption of knowledge automation software called CORVID was added as term 
project. As students in this class are required to learn Expert System, one of the widely used AI paradigms, the 
automation software aims to provide students hands-on experience to apply AI theories in representing, searching 
and capturing intelligence, which is often considered the most important part of any successful AI applications. 
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The initiation of these research-supportive curricula in the Department for undergraduate students’ early 
involvements in research is to enhance students’ professional and personal development, career preparation, and 
general skill development. The Council on Undergraduate Research (CUR) pointed out the growing recognition of 
the value of integrating research and scholarship into the undergraduate curricula include: help students with their 
career choices, develop skills that are not taught in the standard university curriculum, greatly improve 
communication and problem-solving skills, and much more!) 


Ill. Assessment beyond Classroom 


At the classroom level, a range of assessment strategies can be used to understand students’ growth in these 
areas. The direct assessment of students’ course grades were the accumulation grades of assignments, projects and 
exams. Course grades from all four courses were collected. These data, once accumulated for more than one 
semester, will provide very effective assessment to check student in-class performance with the instructor’s 
expectation as well as student learning and project goals. However, it is the indirect assessment activities that not 
only provided students great opportunities for constructive feedback , suggestions for instructional improvement, 
and their learning experiences beyond classroom but also make students feel respected in the process of learning. 

Assessment of competencies beyond classroom is very necessary to provide relevant, specific information 
about undergraduate engineering student learning in the aforementioned vital areas specified by ABET 3a-3k. It is 
possible to assess these competencies at a jurisdictional level, the two level surveys used in this research provided 
quantified information about complex competencies. 


a. RISEC Survey Results 


Two indirect assessments have been strategically incorporated in the project as mentioned earlier. At the 
end of the semester, students were asked to take the RISEC survey to provide feedback specific regarding their 
learning experiences of AI infused curriculum. RISEC survey has three sections; the first section is collecting the 
data about student learning engagement and interaction related to learning activities and assignments; the second 
section is about student learning outcomes related to the research components and the use of the AI tools; and the 
last section is open responses from students regarding the overall class quality and recommendations. The following 
data summarized RISEC data collected from four different level classes in 2018 spring semester: CIVE 141, CIVE 
224, CIVE 339 and CMET 438. The participation rates for this voluntary survey are: 26%, 78%, 32% and 89% 
respectively. In the following tables, N stands for number of participants in the particular class, E stands for total 
enrolled students. Results shown here are the positive (very much, quite a bit) responses. There was no negative 
response received in all four classes, and very small percentages of neutral responses. The neutral and negative 
responses are omitted for brevity. 

Table | through Table 3 summarized selective RISEC questions related to three types of assessment: Table 
1 - Student engagement and interaction in the process of learning; Table 2 - Student research and problem solving 
skills, with emphasis on mental activities; Table 3 - Student feedback on how the artificial intelligence infusion 
project contributed to their broader abilities in and beyond classroom. 

The AI infused curriculum introduced a collection of different AI tools and their applications in real 
engineering applications. These are the situations in which students can make connections between classroom 
activities and reality. This connection creates an optimal learning environment to increase student learning interest 
and drive their motivation that links to their goals in academic achievement and professional career in their future. 
Following observations can be made from student’s feedback In Table 1: 


For freshmen level CE core course (CIVE 141), 100% students reported engaged in learning, and the AI 
infusion not only helped their understanding of the content knowledge, but also increased their interest in future 
research. They also reported integrating ideas, concepts and information from various sources. 

The sophomore level CE core course (CIVE 224) has higher participation rate compared to the other CE 
courses. 100% students reported the implementation improved their understanding of the contents and such also 
increased their interest in the subject matter. 

For the junior/senior CE core course (CIVE 339) engagement to the learning process, better understanding of 
the content and increased interests were evidenced by 100% positive feedback. 

The ECE program junior/senior elective course (CMET 438) has the highest participation rate. All assessment 
questions related to student engagement and interaction received above 70% positive feedbacks. However, it 
lacks unanimously agreed learning gains from students but shows a more even across the board improvement. 
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Table 1. Student Engagement / Interaction Related to Learning Activities and Assignments 


CIVE 141 CIVE 224 CIVE 339 CMET 438 
N=7 | E=27 | N=21 | E=27 | N=6 | E=19 | N=17 | E=19 


Very | Quite | Very | Quite | Very | Quite | Very | Quite 
much | abit | much | abit | much | abit | much | abit 


Selected Assessment Questions 


The activities/assignments in this class 


i ; 43% | 57% | 85% | 14% | 67% | 33.0% | 71% | 12% 
engaged me in the learning process. 


The activities/assignments in this class 
helped me improve my understanding of the 
content knowledge. 


My interest in the subject matter has 
increased due to the activities/assignments 
in this class. 

My interest in future research has increased 
due to the activities/assignments in this 
class. 


I was motivated to do more than the 
minimum requirements in this class. 


I worked on a project that required 
integrating ideas or information from 
various sources. 


I put together ideas or concepts from my 
other courses when completing assignments 
or during class discussions. 


The last five questions in RISEC survey section one ask the coursework emphasis on the mental activities, 
intending to find out if students had increased the ability of knowledge application through analyzing, synthesizing, 
making judgement and applying theories to solving practical problems both in and beyond classroom setting. See 
Table 2 below. Following observations can be made from student’s feedback In Table 2: 


For freshmen level CE core course (CIVE 141), the coursework’s major emphasis on mental activities are 
memorizing facts, ideas or methods, analyzing basic elements and applying theories or concepts to practical 
problems or in new situation100% students reported engaged in learning, and the AI infusion not only helped 
their understanding of the content knowledge, but also increased their interest in future research. They also 
reported integrating ideas, concepts and information from various sources. 

For the sophomore level CE core course (CIVE 224) 100% students reported the emphasis was on analyzing 
idea, experiences or theory. 

For the junior/senior CE core course (CIVE 339) 100% students reported the emphasis lies both on memorizing 
facts, ideas, methods and analyzing basic elements of ideas, experiences or theories. 

The ECE program junior/senior elective course (CMET 438) reported a pretty even emphasis on all five mental 
activities with above 88% positive feedbacks. 


These data indicate that the coursework and learning activities are in alignment of project and teaching objective, 
which is to train students to use knowledge and skills to solve practical problems. 
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Table 2. Coursework Emphasis on Mental Activities 


CIVE 141 CIVE 224 CIVE 339 CMET 438 
N=7 | E=27 | N=21 | E=27 | N=6 | E=19 | N=17 | E=19 


Selected Assessment Questions 
Very | Quite | Very | Quite | Very | Quite | Very | Quite 
much | abit | much | abit | much | abit | much | abit 


Memorizing facts, ideas, or methods from 
your courses and readings so you can repeat 
them in pretty much the same form. 


Analyzing the basic elements of an idea, 
experiences, or theory, such as examining a 
particular case or situation in depth and 
considering its components. 


Synthesizing and organizing ideas, 
information, or experiences into new, more 
complex interpretations and relationships. 


Making judgments about the value of 
information, arguments, or methods, such as 
examining how others gathered and 
interpreted data and assessing the soundness 
of their conclusions. 


Applying theories or concepts to practical 
problems or in new situations. 


One of the main goals of the project is to promote learning beyond classroom by fostering interdisciplinary 
academic settings and to support undergraduate students’ early involvement in research. In the first section, more 
than 83% of the students reported that their research interest increased due to the class activities and assignments. 
More than 82% of the students also reported that the class activities required integrating ideas, information, 
concepts, and theories from various sources out of their own classroom settings. 

The second section of student survey is focusing on collecting data (Table 3) on student research skills 
related to the use of AI tools from their own learning experiences. Following observations can be made from 
student’s feedback In Table 3: 


For freshmen level CE core course (CIVE 141), more than 86% students indicated an overall improvement in 
their abilities in research related competencies. The freshmen students particularly reported 100% positive in 
the following abilities: utilize different knowledge source other than textbook, awareness of the responsible 
conduct of being an engineer/researcher and use new tools/algorithms/software to solve a given problem. 

For the sophomore level CE core course (CIVE 224) more than 86% students reported the implementation of 
the AI infused curricula helped very much or quite a bit on all abilities assessed in the RISEC survey. 
Particularly 100% positive in their ability to develop problem solving plans and articulate research findings 
through written assignments, final products, and/or oral presentations. 

For the junior/senior CE core course (CIVE 339) five out of the ten assessed abilities received 100% positive 
feedback: with the infused AI case studies and term project, students reported their competencies improved very 
much or quite a bit in identifying basic principles and core knowledge, the engagement helped them in making 
connections between this course to other engineering courses, in utilizing different knowledge sources outside 
their textbook and in using new tools/algorithms/software to resolve a particular problem at hand. 

The ECE program junior/senior elective course (CMET 438) reported a pretty even positive improvement in all 
ten abilities assessed with above 82% positive feedbacks. 
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Table 3. Student Feedback on Research Knowledge and Skills Related to the Use of AI Tools 


CIVE 141 CIVE 224 CIVE 339 CMET 438 
N=7 | E=27 | N=21 | E=27 | N=6 | E=19 | N=17 | E=19 


Very | Quite | Very | Quite | Very | Quite | Very | Quite 
much | abit | much | abit | much |] abit | much | abit 


Selected Assessment Questions 


Your ability to “identify basic principles and za | 14% | 57% | 39% | 67% | 33% | 77% | 12% 
knowledge related to core material”. 


Your ability to “make connections between 


: : , 3 43% | 43% | 43% | 48% | 17% | 83% | 53% | 41% 
this course to other engineering courses 


Your ability to “develop a plan to address or 
resolve a specific question or problem.” 


Your ability to “collect and interpret data 
and information in an attempt to resolve the 
question or problem.” 


Your ability to “analyzing different scenarios 
and finding the best solution.” 


Your ability to “trouble shoot your 
solutions” 


Your ability to “utilize different knowledge 
source other than textbook related 
information to solve a question or problem” 


Your “awareness of the responsible conduct 
of being an engineer/researcher.” 


Your ability to “use new 
tools/algorithms/software to solve a 
problem” 


Your ability to “articulate your research 
findings through written assignments, final 
products, and/or oral presentations.” 


The division of courses tends to isolate knowledge from one course to another. Isolated knowledge creates 
a barrier to understand the real nature of the problem. It narrows down the possibilities, opportunities and abilities to 
solve the problems. In reality, everything is interconnected through one way or another. It is extremely important for 
students to make connections of knowledge between different courses and with the real world. AI tools build a 
learning environment that facilitates students to make the connection of knowledge with real world. 

The authors believe AI application infusions are particularly helpful for junior/senior engineering students 
facing complex questions that mimic real world engineering problems such as design, planning, diagnosis, 
classification, management, and decision making scenarios. The RISEC results of AI infused curricula are 
affirmative and very positive in enhancing students learning experiences both in and beyond classroom. 

In the open section of student feedback, ninety-five percent of students think it is important to introduce AI 
knowledge in the course, ninety percent of students reported that AI tools are very efficient in comparison to the 
traditional methods in the process of solving problems. The data analysis from RISEC survey is instrumental to the 
instructional improvement and project expansion. Students provided very positive feedback on AI integration, 
learning activities and improved research skills, which confirmed AI infused curriculum has positive impact not 
only on academic content knowledge but also on broader competencies beyond classroom. 
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b. CURE Survey Results 


The second indirect assessment uses Classroom Undergraduate Research Experience (CURE) national 
survey. CURE survey “grew out of a creative collaboration of faculty from Grinnell College, Hope College, Harvey 
Mudd College, and Wellesley College. The CURE may be used as a pretest-posttest or posttest-only survey to 
measure student experiences in "research-like" or other science courses” (Grinnell College) '!. It is licensed under a 
Creative Commons Attribution-Non Commercial-Share Alike 4.0 International License, copyright 2005-2018 
Grinnell College. The survey has three components, 1) a pre-course survey including demographic questions, 
reasons for taking the course, level of experiences on various course elements, science attitude and learning style 
questions, 2) a post-course survey including estimation of learning gains in the course elements, estimation of 
learning benefits, overall evaluation of the experience and science attitude, and 3) a brief survey for the course. In 
this project, CURE survey is used as a tool to collect bench mark data to compare TSU students with Nation-wide 
students in STEM field, and to measure students learning outcomes related to undergraduate research experiences 
and investigates various elements related to student learning involving the use of AI tools. 

The CURE surveys were conducted in Spring 2018 semester to three CE courses: freshmen level CIVE 
141, sophomore level CIVE 224 and junior/senior level CIVE 339. The pre-survey was conducted before AI infused 
content was introduced and post-survey was conducted during the last week of the semester, after all curriculum 
content was completed. Similar positive results were reported from all three courses. For brevity, reported data 
collected from junior/senior level course CIVE 339 are presented here with a total enrollment of 19 students, 14 
male students and 5 female students and 95% of them are seniors. Students participated in both pre-course and post- 
course surveys voluntarily. All students participated in both pre and post surveys. For the sections of course 
elements, course benefit, and attitudes toward science, CURE analyst provided TSU with mean data. So, the data 
analysis method of Independent-Samples T Test is used to compare TSU mean results with that of national data to 
find out statistical significance with the confidence level set at 95%. 

In the pre-course survey, there is a question of ten reasons to take the course for students to rank. The top 
six reasons for TSU students to take the course are listed in Table 4, which indicate student’s expectation for the 
course. Besides needing the course to fill a major requirement, students wanted to learn about science and research 
process through hands-on research experience for desired employment after college. 


Table 4. Top Six Very Important Reasons to Rake the Class 


Reasons to take the course 


I need it for graduate or professional school. 


To fill a distribution requirement 


I need it for my desired employment after college. 


To fill a requirement for my major 


To learn about science and the research process 


To get hands-on research experience 


In the pre-course survey, students were asked to assess their prior experience on 25 course elements, and in 
the post-course survey, students were asked the same course elements to rate their gains they have made as a result 
of taking the course. TSU students showed their improvements in 21 out of 25 course elements, a coverage of 84% 
overall, which is higher than the Nation-wide results of 19 out of 25 course elements, which is 76%. The mean 
comparison result from Independent-Samples T Tests between TSU pre-course experience and post-course gains 
generates a P=0.000, which is statistically extremely significant, see Table 5 below. When compare TSU students’ 
post-course gains with nation-wide post-course gains, again, the Independent-Samples T Tests generates P=0.000, 
see Table 6 below. 
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Table 5. Mean Comparison of TSU Pre-course Experience and TSU Post-course Gains 


Sig. 
(2-tailed) 


Groups Elements Std. D. 


TSU Pre-course 
Experience 


TSU Post-course 
Gains 


25 0.16169 


25 0.16362 


Table 6. Mean Comparison of TSU Post-course Gains and Nation-wide Post-course Gains 


Sig. 


Groups Elements Std. D. (2-tailed) 


TSU Post-course 


Caine 25 0.16362 


Nation-wide 
Post-course 0.27588 
Gains 


The top six post-course gains reported by TSU students are listed below. These knowledge and skills are 
directly associated with student ability to conduct research through the use of computer technology and to produce 
research results. 


A project where students have input into process or topic 
A project entirely of student design 

Work on problem sets 

Computer modeling 

Present results in written papers or reports 


Altea aa 


There are twenty-one items that are measured in the post-survey for learning benefits from the course. 
TSU students had 17 (81%) items scored higher than Nation-wide data. The Independent-Samples T Test generated 
P=0.000, which is statistically extremely significant. See Table 7 below. 


Table 7. The Mean Comparison of Course Benefits between TSU Students and Nation-wide Students 


Sig. 


Groups Std. D. (2-tailed) 


TSU 0.14427 
Nation-wide 0.20822 


0 


The top six benefits reported by TSU students from the course are as listed below. AI infusion not only 
prepares students to be technologically competitive in either careers, but also increases students’ self-confidence that 
will help them achieve greater successes in all aspects of their life. 


Clarification of a career path 

Confidence in my potential to be a teacher of science 
Ability to read and understand primary literature 
Learning to work independently 

Skill in how to give an effective oral presentation 
Self-confidence 


ON ov ects oS 


In the section of science attitudes, students were asked 22 questions in both pre-course and post-course 
surveys. TSU students reported 17 (77%) items with positive changes. In mean comparison of TSU student pre- 
course to post-course science attitude, The Independent-Samples T Test generated P=0.014, which is statistically 
significant. See Table 8 below. This course has enhanced students’ concept of science knowledge and also further 
strengthened their science attitudes. 
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Table 8. The Mean Comparison of TSU Student Science Attitude Change in Pre and Post-course Surveys 


Groups Std. D. Sig. (2-tailed) 


Pre-course 
Science Attitude 


0.35626 
0.014 


In the post-course survey, there are four questions that serve as an overall assessment of the course. The 
following table 9 shows the results of TSU students’ rating for the course. 


Table 9. Overall Course Rating by TSU Students 


Rating Questions Strongly Agree 


This course was a good way of learning about the subject 83% 


This course was a good way of learning about the process of scientific research 84% 


This course had a positive effect on my interest in science 80% 


I was able to ask questions in this class and get helpful responses 84% 


The CURE survey results from TSU students are very affirmative and positive. Students expressed their 
expectation of taking this course, which is to learn about science and research process through hands-on research 
experience for desired employment after college. In the survey, they provided strong evidence that AI infusion did 
impact their knowledge acquisition and research ability improvement demonstrated in their report in learning 
experience enhancement, knowledge gains and course benefits. AI infusion is a very well-timed project that prepares 
TSU students with the skills and knowledge for their future professions. US Bureau of Labor Statistics reported that 
“more than 50% of today’s jobs require some degree of technology skills, and experts say that percentage will 
increase to 77% in the next decade” and “Employment of computer and information technology occupations is 
projected to grow 13 percent from 2016 to 2026, faster than the average for all occupations”!, Through hands-on AI 
projects, TSU students have been equipped with the research knowledge and problem-solving skills to be 
competitive in the job market. 
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Abstract 


Indirect assessments have not caught the attention of many educators. Most instructors focus on using direct 
assessments to measure student learning outcomes. However, effective use of indirect assessments can provide 
instructors with valuable information to make teaching improvement based on student feedback. This paper will 
present a research project that implemented multi-dimensional assessment activities to evaluate student-centered 
learning. The analyzed indirect assessment data have been used strategically to enhance teaching quality and 
curriculum enrichment, to improve student learning as well as for research program expansion. 


Keywords: Direct assessment, indirect assessment, student-centered learning, learning engagement, student 
feedback, Artificial Intelligent, civil engineering education 


Introduction 


Texas Southern University (TSU) is located at Houston, the fourth largest city in US. TSU is ranked as the 
second nation’s largest HBCU (Historically Black College and University) by enrollment with 12 colleges and 
schools and offering more than 100 undergraduate and graduate programs and concentrations. College of Science, 
Engineering & Technology has the largest enrollment every year and educates students with knowledge and skills 
for the job market. The National Science Foundation created a grant program HBCU-UP (Historically Black 
Colleges and Universities Undergraduate Program) to enhance the quality of science, technology, engineering and 
mathematics (STEM) instructional and outreach programs at HBCUs as a means to broaden participation in the 
Nation’s STEM workforce. TSU civil engineering program has been awarded this grant for four years (2015-2019) 
to design an artificial intelligence project. The project seeks to infuse innovative Electrical/Computer Engineering 
specialized Artificial Intelligence (AJ) tools into traditional Civil Engineering problem-solving routines through 
problem-based learning approach (PBL). It helps bridge current curricula gap in the Department of Engineering at 
TSU. The objectives of the project are: 1) to develop an intelligent knowledge database to document, compare, and 
analyze cutting-edge AI applications in civil engineering field, which can be used as the platform and educational 
media for curricula development and implementation for PBL approaches in classroom; 2) to add one new 
interdisciplinary course to the Department’s curricula “AI Tools for Engineering Problem Solving” for all senior 
engineering major students; 3) to enrich current curricula by integrating innovative AI application case studies into 
more than fourteen existing courses being offered in the Department, 4) to foster interdisciplinary academic setting 
by hosting server-based intelligent database in the College of Science, Engineering and Technology and to provide 
web- and classroom-based workshops and tutorials to all interested students and faculty; and 5) to support 
undergraduate students’ early involvement in research. 

In the project implementation, besides developing hands-on projects and research activities through the use 
of AI tools, a series of assessment activities were designed to make sure students are learning what project designed 
for them to learn and to find out the impact of the project on student learning. Student learning is affected by 
different factors. It is very essential to collect precise and wide-ranging assessment data from difference sources in 
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order to make effective improvements in course design and instructional quality to enhance student learning. A good 
assessment practice shifts the focus from blaming students for not achieving expected learning outcomes to finding 
out true reasons behind student learning issues. The grant project implemented multi-dimensional assessments. 
Student learning outcomes are evaluated through direct assessments in course assignments; projects, group 
activities, quizzes and exams, and student grades have been used to analyze student class performance at the same 
time to provide meaningful information for course improvements. In addition to the direct assessments in class, two 
indirect assessments have been conducted to collect data on student learning from their perspective. Both indirect 
assessments have been using survey method. One was developed in-house, the Research on the Integrated Science 
and Engineering Curriculum (RISEC). This survey is specifically targeting at student learning and using the AI tools 
in their hands-on projects and research activities to find out 1) student engagement and interaction in the process of 
learning, 2) student research and problem solving skills, 3) student feedback on how the artificial intelligence 
infusion project contributed to their learning, and 4) student open feedback on overall effectiveness of instructional 
delivery and project implementation. The second survey is CURE, the Classroom Undergraduate Research 
Experience (CURE) survey, which has been used nation-wide to measure student experiences in "research-like" or 
other science courses. In this project, CURE data is used as a bench mark to compare TSU students with Nation- 
wide students in STEM field in science attitude, learning experiences, learning gains, learning benefits and overall 
evaluation of the course. The survey results give the instructor an opportunity to find out the course teaching 
effectiveness, and more importantly to compare TSU students with national counterparts in academic performance. 
The additional value of these surveys is to find out the impact of AI infusion on student learning and their future 
career. The indirect assessment data contribute to the project’s success in several aspects. Firstly, the data have been 
used to make instructional improvement, secondly, student feedback provided very important information for AI 
project expansion, thirdly the data help the instructor to compare the performance of TSU students with national 
peer in STEM field, and lastly, the data provide evidence of the impact of AI tools on student learning. The data are 
also very effective in checking the alignment of student performance with the instructor’s expectation as well as the 
alignment of student learning and project goals. Effective indirect assessment activities make students feel respected 
in the process of learning in providing suggestions for instructional improvement. Students are truly involved in the 
process of teaching and learning with constructive feedback and their learning experiences. 


Student Learning Assessments 


“Assessing what really matters to student learning” or to instructor’s teaching, or both? There are still a lot 
of instructors who teach and assess their students in the same way as they were taught and assessed when they were 
students in schools, mostly focusing on memorized knowledge. Since mid-80s assessment movement, demanding to 
know how students learn in school, mainly driven by accreditation requirements, more and more attention has been 
paid to use assessment data for the improvement of student learning. However, largely, the good practice has not 
been implemented in schools due to lack of resources, such as proper training of faculty to design and teach 
effectively, understanding modern challenges on educational system, few of teachers “have had formal instruction 
on how to do it and do it well. Even fewer of us (teachers) have had training on how to provide evidence of what 
students are learning in ways that are acceptable to external stakeholders” (Barkley & Major, 2015, p1). In addition, 
with increased number of enrollment, teaching process has been trimmed to the minimum via online automated 
tools. The feedback and interaction from both sides, instructors with students, and students with students, are 
disappearing in this automation. The classroom assessments have become a simple tool to sign grades to students 
and label them with levels. Assessment results are seldom used for professional development and instructional 
improvement. In order to solve this problem and truly improve student learning, teachers need to be more systematic 
and sensitive observers of learning and more efficiently and effectively identify what is important for students to 
learn, and implement appropriate activities to ensure that students learn it (Barkley & Major, 2015). Carnegie 
Mellon Eberly Center for Teaching Excellence and Education Innovation provides seven teaching principles. Three 
of them highlight the importance of understanding of students, and use reflection and feedback for professional 
progress. “Effective teaching involves acquiring relevant knowledge about students and using that knowledge to 
inform our course design and classroom teaching. Effective teaching involves recognizing and overcoming our 
expert blind spots. Effective teaching involves progressively refining our courses based on reflection and feedback” 
(Teaching Principles, no date). These three principles stress on course design based on student knowledge, 
background information and needs and utilize student feedback and instructor’s reflection for course improvement. 
“Without knowing how students spend their time, it’s almost impossible to link student learning outcomes to the 
educational activities and processes associated with them” (Kul, 2001, p15). The more teachers know and 
understand students, the more efficient and effective the teaching will be. 
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Assessment, in different format, is the only way to evaluate what and how student learn and how well they 
learn. Both direct and indirect assessments are effective methods to collect data for this purpose. Direct assessments 
provide for the direct examination or observation of student knowledge or skills against measurable learning 
outcomes (Roger, 2006). These assessments will provide data on student knowledge learned and grasped in 
classroom activities — what students learn. While, indirect assessment is a strong supplement to direct assessment in 
giving teachers information on how students learn from their perspectives and experiences, assessing what really 
matters to student learning (Kuh, 2001). “Indirect assessments of student learning ascertain the perceived extent or 
value of learning experiences. They assess opinions or thoughts about student knowledge or skills. Indirect measures 
can provide information about student perception of their learning and how this learning is valued by different 
constituencies” (Roger, 2006). The importance of assessments has been undermined by most teachers because most 
of them are using assessment to evaluate what have been taught to students by simple assigning grades without 
thorough understanding the true meaning of assessments. Student grades have rarely been analyzed to find out the 
real problems of student learning issues. Mostly, the teachers tend to blame students for lack of efforts in learning 
instead of connecting assessments with instructional alignment and teaching practice. The best assessments are those 
whose results are used to inform meaningful, important decisions (Suskie, 2018). 


RISEC Survey 


In this project, direct assessments were designed to fully engage students in hands-on activities to practice 
and explore the real world case studies and learn the knowledge and skills to solve problems. Since the AI tools are 
newly developed and integrated in the teaching and learning process, it is extremely important to get feedback from 
student learning experiences to further improve the quality of the tools and also get to know the impact of the project 
on student learning. Two indirect assessments have been strategically incorporated in the project as mentioned 
earlier. At the end of the semester, students were asked to take the RISC survey to provide feedback specific 
regarding the implementation of AI tools. RISEC survey has three sections; the first section is collecting the data 
about student learning engagement and interaction related to learning activities and assignments; the second section 
is about student learning outcomes related to the research components and use of AI tools; and the last section is 
open responses from students regarding the overall project quality and recommendations. The following data were 
collected from a sophomore level civil engineering class in spring 2018 semester. The participation rate is 78%, 
twenty-one out of twenty-seven students took the survey. The following table 1 shows the data results from the 
participants for the first section in the survey. 


Table 1. Learning Engagement / Interaction Related to Learning Activities and Assignments 


No Question bed ae 7) Some Ene 
1 oe ee in this class engaged me in the learning 85% 15% 0% 0% 
2 The activities/assignments in this class helped me improve my 16% 24% 0% 0% 
understanding of the content knowledge. 

3 My interest in the subject matter has increased due to the 62% 38% 0% 0% 
activities/assignments in this class. 

4 My interest in future research has increased due to the 67.8% | 23.8% | 9.5% 0% 
activities/assignments in this class. 

5 : i motivated to do more than the minimum requirements in this 57% 38% 5% 0% 

6 I worked on a project that required integrating ideas or information 62% 33% 5% 0% 
from various sources. 

7 I put together ideas or concepts from my other courses when 48% 38% 10% 50/ 
completing assignments or during class discussions. 


From student report, 100% of them think the learning activities and assignments have helped them either 
very much or quite a bit in engaging their learning process, in helping improve their understanding of the content 
knowledge and increased their interest in the subject matter. 95% of students think either very much or quite a bit 
that they were motivated to do more than minimum requirements in the class and they integrated ideas and 
information from other resources when they worked on the project. 90% of them reported either very much or quite 
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a bit that their interest in future research had increased due to the activities and assignments in the class. AI Tools 
are a collection of real world study scenarios, which are the situations that are happening in real life around them. 
When students study and practice those cases for knowledge acquisition, they see the connection between classroom 
activities and reality. This connection creates an optimal learning environment to increase student learning interest 
and drive their motivation that links to their goals in academic achievement and professional career in their future. 

The last five questions in section one focus on the coursework emphasis on the mental activities, intending 
to find out if students had increased the ability of knowledge application through analyzing, synthesizing, making 
judgement and applying theories to solving practical problems, see Table 2 below. 


Table 2. Coursework Emphasis on Mental Activities 


. Very | Quitea Very 
No Question aan bit Some little 
1 Memorizing facts, ideas, or methods from your courses and readings 29% 67% 50% 0% 
so you can repeat them in pretty much the same form. 
Analyzing the basic elements of an idea, experiences, or theory, such 
2 | as examining a particular case or situation in depth and considering its 52% 48% 0% 0% 
components. 
Synthesizing and organizing ideas, information, or experiences into fe . ‘ ‘ 
3 52.4% | 42.9% 0% 0% 


new, more complex interpretations and relationships. 


Making judgments about the value of information, arguments, or 
4 | methods, such as examining how others gathered and interpreted data | 42.9% | 47.6% 5% 0% 
and assessing the soundness of their conclusions. 


Applying theories or concepts to practical problems or in new 61.9% | 33.3% 10% 0% 
situations. 


100% of students reported either very much or quite a bit that the coursework emphasize on analyzing, 
synthesizing and organizing ideas and information. 95% of students think very much for quite a bit that the course 
emphases on making judgements and 90% of them think the coursework emphasizes on applying theories or 
concepts to practical problems. These data indicate that the coursework and learning activities are in alignment of 
project and teaching objective, which is to train students to use knowledge and skills to solve practical problems. 

One of the objectives of the grant project is to support undergraduate students’ early involvement in 
research. In the first section, seventy-five percent of students reported that their research interest increased due to the 
class activities and assignments. The second section of student survey is focusing on collecting data (Table 3) on 
student research skills related to the use of AI tools from their own learning experiences. 


Table 3. Student Feedback on Research Knowledge and Skills Related to the Use of AI Tools 


N Please rate how this course has contributed to your improvement in Very | Quitea Very 
te) : ? Some : 
the following areas: much bit little 
1 oe seed i identify basic principles and knowledge related to 579% 38% 50/ 0% 
> Your ability to make connections between this course to other 42.9% | 47.6% | 9.5% 0% 
engineering courses 
3 Your ability to develop a plan to address or resolve a specific 52% 48% 0% 0% 
question or problem. 
4 Your ability to “collect and interpret data and information in an 57% 38% 50% 0% 
attempt to resolve the question or problem. 
5 bate et to “analyzing different scenarios and finding the best 52% | 33.3% 15% 0% 
6 | Your ability to “trouble shoot your solutions” 42.9% | 47.6% 5% 5% 
7 Your ability to utilize different knowledge source other than 53% 33% 14% 0% 
textbook related information to solve a question or problem 
8 Your awareness of me responsible conduct of being an 33% 62% 50% 0% 
engineer/researcher. 
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9 Your ability to “use new tools/algorithms/software to solve a 62% 33% 50% 0% 
problem 

10 Your ability to “articulate your research findings through written 71% 29% 0% 0% 
assignments, final products, and/or oral presentations. 


Ninety-five percent of students think that this course has contributed very much or quite a bit to improve 
their ability to identify basic principles and knowledge related to core material, to collect and interpret data and 
information in an attempt to resolve the question or problem, and to increase their awareness of the responsible 
conduct of being an engineer and researcher. Ninety percent of student reported that the course has contributed very 
much or quite a bit to make connections between this course to other engineering courses, and use new 
tools/algorithms/software to solve a problem. The division of courses tends to isolate knowledge from one course to 
another. Isolated knowledge creates a barrier to understand the real nature of the problem. It narrows down the 
possibilities, opportunities and abilities to solve the problems. In reality, everything is interconnected through one 
way or another. It is extremely important for students to make connections of knowledge between different courses 
and with the real world. AI tools build a learning environment that facilitates students to make the connection of 
knowledge with real world. 

In the open section of student feedback, ninety-five percent of students think it is important to introduce AI 
knowledge in the course, ninety percent of students reported that AI tools are very efficient in comparison to the 
traditional method in the process of solving problems. The data analysis from RISC survey is instrumental to the 
instructional improvement and project expansion. Students provided very positive feedback on AI integration, 
learning activities and improved research skills, which confirmed the impact of AI infusion on student learning. 


CURE survey 


The second indirect assessment uses Classroom Undergraduate Research Experience (CURE) national 
survey. CURE survey “grew out of a creative collaboration of faculty from Grinnell College, Hope College, Harvey 
Mudd College, and Wellesley College. The CURE may be used as a pretest-posttest or posttest-only survey to 
measure student experiences in "research-like" or other science courses” (Grinnell College). It is licensed under a 
Creative Commons Attribution-Non Commercial-Share Alike 4.0 International License, copyright 2005-2018 
Grinnell College. The survey has three components, 1) a pre-course survey including demographic questions, 
reasons for taking the course, level of experiences on various course elements, science attitude and learning style 
questions, 2) a post-course survey including estimation of learning gains in the course elements, estimation of 
learning benefits, overall evaluation of the experience and science attitude, and 3) a brief survey for the course. In 
this project, CURE survey is used as a tool to collect bench mark data to compare TSU students with Nation-wide 
students in STEM field, and to measure students learning outcomes related to undergraduate research experiences 
and investigates various elements related to student learning involving the use of AI tools. 

The following reported data were collected in spring 2018 from civil engineering class CIVE339 and civil 
technology class CIVT337 with a total enrollment of 19 students, 14 male students and 5 female students and 95% 
of them are seniors. Students participated in both pre-course and post-course surveys voluntarily. All students 
participated in both pre and post surveys. For the sections of course elements, course benefit, and attitudes toward 
science, CURE analyst provided TSU with mean data. So, the data analysis method of Independent-Samples T Test 
is used to compare TSU mean results with that of national data to find out statistical significance with the confidence 
level set at 95%. 

In the pre-course survey, there is a question of ten reasons to take the course for students to rank. The top 
six reasons for TSU students to take the course are listed in table 4, which indicate student’s expectation for the 
course. Besides needing the course to fill a major requirement, students wanted to learn about science and research 
process through hands-on research experience for desired employment after college. 


Table 4. Top Six Very Important Reasons to Rake the Class 


No. Reasons to take the course % 
1_| I need it for graduate or professional school. 65% 
2 | To fill a distribution requirement 63% 
3 | I need it for my desired employment after college. 61% 
4 | To fill a requirement for my major 58% 
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5 | To learn about science and the research process 58% 


6 | To get hands-on research experience 53% 


In the pre-course survey, students were asked to assess their prior experience on 25 course elements, and in 
the post-course survey, students were asked the same course elements to rate their gains they have made as a result 
of taking the course. TSU students showed their improvements in 21 out of 25 course elements, a coverage of 84% 
overall, which is higher than the Nation-wide results of 19 out of 25 course elements, which is 76%. The mean 
comparison result from Independent-Samples T Tests between TSU pre-course experience and post-course gains 
generates a P=0.000, which is statistically extremely significant, see Table 5 below. When compare TSU students’ 
post-course gains with nation-wide post-course gains, again, the Independent-Samples T Tests generates P=0.000, 
see Table 6 below. 


Table 5. Mean Comparison of TSU Pre-course Experience and TSU Post-course Gains 


Groups Elements Mean Std. D. t df Sig. (2-tailed 
TSU Pre-course Experience 25 3.6944 16169 = -5.999 48 0.000 
TSU Post-course Gains 25 3.9704 .16362 


Table 6. Mean Comparison of TSU TSU Post-course Gains and Nation-wide Post-course Gains 


Groups Elements | Mean Std. D. t df Sig. (2-tailed) 
TSU Post-course Gains 25 3.9704 .16362 5.525 48 0.000 
Nation-wide Post-course Gains 25 3.6160 27588 


The top six post-course gains reported by TSU students are listed below. These knowledge and skills are 
directly associated with student ability to conduct research through the use of computer technology and to produce 
research results. 

1. A project where students have input into process or topic 
A project entirely of student design 
Work on problem sets 
Computer modeling 
Present results in written papers or reports 


Ou ge Go 


There are twenty-one items that are measured in the post-survey for learning benefits from the course. 
TSU students had 17 (81%) items scored higher than Nation-wide data. The Independent-Samples T Test generated 
P=0.000, which is statistically extremely significant. See Table 7 below. 


Table 7. The Mean Comparison of Course Benefits between TSU Students and Nation-wide Students 


Groups N Mean Std. D. t df Sig. (2-tailed) 
TSU 21 3.8914 0.14427 5.195 40 0.000 
Nation-wide 21 3.6043 0.20822 


The top six benefits reported by TSU students from the course are as listed below. AI infusion not only 
prepares students to be technologically competitive in either careers, but also increases students’ self-confidence that 
will help them achieve greater successes in all aspects of their life. 

1. Clarification of a career path 
Confidence in my potential to be a teacher of science 
Ability to read and understand primary literature 
Learning to work independently 
Skill in how to give an effective oral presentation 
Self-confidence 


Dv de 


In the section of science attitudes, students were asked 22 questions in both pre-course and post-course 
surveys. TSU students reported 17 (77%) items with positive changes. In mean comparison of TSU student pre- 
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course to post-course science attitude, The Independent-Samples T Test generated P=0.014, which is statistically 
significant. See Table 8 below. This course has enhanced students’ concept of science knowledge and also further 
strengthened their science attitudes. 


Table 8. The Mean Comparison of TSU Student Science Attitude Change in Pre and Post-course Surveys 


Groups N Mean Std. D. t df Sig. (2-tailed) 
Pre-course Science Attitude 22 3.5945 0.35626 -2.551 42 0.014 
Post-course Science Attitude 22 3.8509 0.76261 


In the post-course survey, there are four questions that serve as an overall assessment of the course. The 
following table 9 shows the results of TSU students’ rating for the course. 


Table 9. Overall Course Rating by TSU Students 


No. Rating Questions Strongly Agree 
|__| This course was a good way of learning about the subject 83% 
2__| This course was a good way of learning about the process of scientific research 84% 
3__| This course had a positive effect on my interest in science 80% 
4 | I was able to ask questions in this class and get helpful responses 84% 


The CURE survey results from TSU students are very affirmative and positive. Students expressed their 
expectation of taking this course, which is to learn about science and research process through hands-on research 
experience for desired employment after college. In the survey, they provided strong evidence that AI infusion did 
impact their knowledge acquisition and research ability improvement demonstrated in their report in learning 
experience enhancement, knowledge gains and course benefits. AI infusion is a very well-timed project that prepares 
TSU students with the skills and knowledge for their future professions. US Bureau of Labor Statistics reported that 
“more than 50% of today’s jobs require some degree of technology skills, and experts say that percentage will 
increase to 77% in the next decade” and “Employment of computer and information technology occupations is 
projected to grow 13 percent from 2016 to 2026, faster than the average for all occupations” (Aril, 2018). Through 
hands-on AI projects, TSU students have been equipped with the research knowledge and problem-solving skills to 
be competitive in the job market. 


Conclusion 


Involving students in the process of course design and instructional delivery is very important for efficient 
and effective teaching practice. When instructors understand student’s needs and the factors that affect students 
learning, instructors can consciously pay attention to those factors and properly address them through different 
teaching strategies. In the grant project, due to the effective use of both direct and indirect assessments, the course 
instructor used the analyzed data make curriculum and instructional improvement to expand AI infusion to benefit 
more students. Effectively integrating student feedback in the teaching process creates a genuine student-centered 
learning because the course design and delivery are truly around students and for the students. Most student-centered 
learning is focusing on learning activities that are conducted by students with the guidance of instructors, very rarely 
including student learning factors in the process of course design and delivery. Fink pays attention to students to 
determine what they believed were truly significant learning experiences that changed the way they lived their 
personal, social, civic, or professional lives (2013). Direct assessments are important to evaluate student learning 
outcomes, however indirect assessments can be used effectively to better understand student needs and know more 
about their learning challenges, which can help instructor design a true student-centered learning and improve 
overall education quality. 
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Abstract 


The term Social Group refers to groups made up of at least two people, who work together in an activity 
system. These people will also share the same objectives and common behaviors, which are under the rules defined 
by those groups. Activity theory as a social psychological perspective is used to state the supportive cycle in 
interpersonal relationships. These relationships occur as the group where the communication of its members is 
influenced by the structure of said group. A reflective e-portfolio can be used as a tool that looks to develop both 
oneself, and the achievements of the group. Revealed in this study, are the effects on self-efficacy skills in career 
decision making when using a reflective e-portfolio in a large and small social support groups. Students from grade 
11, and from various parts of Thailand, were selected as the sample group. These students also attended the summer 
camp in Chulalongkorn University, Bangkok. The total size of the group was 80 students, and they were divided into 
a control (large group), and an experimental group (small group) equally through the method of purposive sampling. 
The data was then statistically analyzed by mean, standard deviation, and t-test. It was found in this study that the self- 
efficacy skills in career decision making in a small social support group was significantly higher than in a large group, 
at the significance level of .05 (p=.02) M= .02. However, the data have not found a significant difference at the level 
of .05, in a large social support group. 


Keywords: Reflective E-Portfolio, Group Size, Activity Theory, Social Support, Career Development, E-Portfolio 
1. Introduction 


Many countries throughout the world are facing a free and boundless global economy under the new 
challenges presented by globalization. Citizens should be able to adapt themselves to these changes by acquiring high 
quality knowledge, abilities, skills and desirable attributes. They can change and develop in order to maximize their 
aptitude skills and levels of interest in a subject, as well having the ability to work and live happily with other people 
in a society. The National Education Plan of Thailand (2017-2036), created by the Secretariat of the Council of 
Education (2017), emphasized the importance of sharing information, communicating technological development, and 
the production and development of human resources, as well as research, through innovation. Boosting the 
competitiveness of the country is what is aimed for when trying to improve these aspects of society. The development 
of guidance systems, to be used in schools, was also outlined in this policy. The purpose of the guidance system is to 
promote self-efficacy in career decision making for the people of all ages. To encourage the development of these 
guidance systems in schools, e-portfolio could be used. The planned outcome is that a decision in relation careers can 
be made, and that careers can be organized into fields that learners are interested in. A new admission system for 
higher education, which was announced by the TCAS Admission System, to enable a student portfolio to be submitted 
to higher education institutions. Matched with this system, candidates can be selected for the first round of acceptance, 
starting from 2018 onwards. 


Utilizing tools to allow learners to reach their full potential, reflective practice is a good way for them to 
understand and evaluate themselves. This can be done with the use of an e-portfolio, but feedback on reflective 
approaches can be limited. Coordinating support from the relevant agencies, such teacher and peers, while completing 
social support activities can help in achieving these outcomes. A large amount of research has been completed on 
group support in relation to academic achievement and self-efficacy. However, work remains to be done regarding 
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with the effects that different group sizes can have. It is possible that group size is one of the factors which affects the 
supportive cycle. The purpose of this study was to observe the effects of group size on support given, in both a large 
and small group setting, during reflective e-portfolio development. We used activity theory to define the roles in the 
community, roles such as which support group played a part in the division of labor while completing tasks. The e- 
portfolio served as a mediating artifact, and self-efficacy skill in career-based decision making was the objective used 
to achieve an outcome in career development. The application of this research may lead to a better understanding of 
the effects of group size on e-portfolio development within a community. 
1.1 Career Development 


Adolescents take on the career developmental task of developing their personalities from school-to-work 
transition, in order to join the workforce. There are 5 stages of which the career developmental tasks are comprised, 
these are: growth, exploration, establishment, management, and disengagement. This means that adolescents need to 
concern themselves with their ability to adapt to different careers, being able to make the right decisions in order to 
control their careers, being curious by reflecting on exploration that they have done, seeking out new information and 
looking to expand their own knowledge, and lastly by having confidence in their career through the consideration of 
their own self-esteem and self-efficacy (Brown & Lent, 2012). Self-Efficacy, which is related to one of the career 
development variables, is defined as self-confidence in behavior, thinking, and emotional expression. Bandura (1997), 
described self-efficacy skills that affect career-based decision-making by stating that most people often avoid making 
career decisions due to uncertainty about themselves. This indicates that making a career decision is not just about 
choosing a career, but also about problem-solving skills, which are likely developed when facing unpredictable 
problems. 


1.2 Social Support Groups 


It is necessary to support learners during transitional periods. Smith (2010, p. 3) reported that the self-efficacy 
social support process is carried out to prepare learners for the Higher Education system. This is done by connecting 
learners to a peer-mentored and peer-facilitated counseling system, encouraging learners to participate in programs or 
camps during their transition, having learners talk in groups to build the confidence in their own abilities, as well as 
stimulating them for new challenges. There are three types of social support, Thiots (2011, 
p. 53) divided these into the following categories: 1) Socioemotional Aid, which encompasses the delivery of love, 
care, vision, compassion and developing a sense of belonging to society. 2) Instrumental Aid which is provided by 
actions or tools so that the role of the sponsors can be fulfilled, an example of this would be providing equipment or 
financial support. 3) Informational Aid is an information-based support type, in the form of facts and opinions, or 
providing feedback. These are beneficial to the situations experienced by the students in real life. Social support was 
described by Symonds (2015, pp. 169-173), as one of the aspects that contribute to the promotion of learners’ well- 
being during transitional periods. Social Support creates a personal profile for leaners, while using social activities in 
group support creates values in relationships, like a desire to meet new friends. Students also learn how to organize 
and participate in programs. 


Although group support showed significance between members of the social group, some factors had to be 
considered in order to first form the support group. According to Cho et al. (2016), different peer group sizes can have 
an effect on basic life support and the feedback process, as such, the experiment was divided into two groups: The 
Standard Group of 4 members, and the Large Group of 8 members. It was shown in the results that the two groups 
had no difference in the post-training scores. However, the standard group had a higher feedback level and most 
members in large groups would have benefitted from a smaller group size. Small groups may have the advantage of 
increased interaction amongst their members. A free exchange of ideas can help to make decisions within groups more 
easily, and involve members in the decision-making process. Despite this, conflicts may arise in smaller sized groups, 
both in the group and also between different groups. In contrast, big groups can access, and make use of, a larger 
amount of resources. It appears that formal interaction patterns and a higher level of strictness when it comes to time 
allocation is in place in the larger groups. One limitation of the large group is that the group members may not 
appreciate each other's abilities due to the fact that the work is carried out as a group task (Kehoe, 2013). 


1.3 Reflective E-Portfolio 


The E-portfolio is an explicit tool used for learning which not only shows the product of learning, but 
processes taken by the students as they completed their work. Learners were helped to make connections between new 
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and previous learning experiences through a systematic collection of works and reflecting on their own backgrounds. 
One of the methods, called the reflective e-portfolio, is the combination of the portfolio approach, along with a 
reflective thinking process. This is done via an electronic platform, which aided in developing the learner’s writing 
skills, while creating links to deep learning and an understanding of self. The processes of the reflective electronic 
portfolio used in this research were divided into 5 steps, which are as follows: 1) Collection of inventions created 
throughout the year, or direct experience in the field. This helps with the learning, developing and acquiring new 
knowledge. 2) Selection of specific works or artifacts from a wider collection, as seen in Step 1, to evaluate and select 
the best evidence of growth. 3) Reflection of events, through telling stories of their own experiences, and outlining 
how they have contributed to their own learning. 4) Evaluation of both formative and summative assessment from 
various stakeholders. 5) Taking Action and Connecting ideas between reflection and evidence, or a piece of work to 
show what they have learned. The aim of this is to create their own example of the learning process, and practice 
thinking systematically about learning experiences. 


1.4 Activity Theory 


Activity Theory is an interdisciplinary concept which consists of philosophical, sociological, culturological, 
psychological and physiological aspects. An example of this would be the use of a psychological rationale basis in 
education, organizations, etc. Another instance would be the societal phenomena issue in economics, cultural studies, 
and political science (Blunden, 2010). Activity Theory is defined as the regulation of institutional activities between 
the participants. Other cultural tools such as the of defining time and space for learners and teachers, between learners 
themselves, and between school and external environments are also included in Activity Theory. This is all based on 
a system of structuring social relations (Daniels, Edwards, Engestrém, Gallagher, & Ludvigsen, 2013). In order to 
study the Activity Theory, its dimensions should be divided into two characteristics, both the study of the individual's 
learning goals, and the study of the belief that the results of learning come from the individuals who are actually 
learning together, asking questions and testing each other within the activity system. It was proposed by Engestrém 
(1999), that the elements of activity theory included subjects, objectives, communities, mediating artifacts, rules, 
divisions of labor and outcomes. In this research, the activity structure was clarified by defining the subjects as the 
students in grade 11, who were all active in the community. Next, the final outcome was defined as the planning 
stage of career development, and the objective or expected dependent variable was defined as the career-based 
decision-making self-efficacy of the students. In the same vein, the e-portfolio was used as the mediating artifact, 
and the social support group used as the division of labor within the community. 


Mediating Artifacts 
Reflective E-Portfolio 


Subjects Object 
Students under Career Decision Outcomes 
the royal patronage \ Self-Efficacy . Career Planning 
in Grade 11 Skills 
Rules Community Division of Labor 
- Attened the support group - Students under the royal patronage ~ Sub-Social Support Group 
- E-Portfolio evaluations - The Financial Aid & Scholarship Office 
- Funding conditions - Supervisors 


1 


Figure | The Activity structure division classified for this research 
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2 Methods 


The purposes of this study were to compare the self-efficacy skills in career decision-making before and after 
using reflective e-portfolio, and to identify the effects of a large and small social support group sizes on reflective e- 
portfolio development. 


2.1 Research design 


Contained in the Quasi Experimental Research were | experiment group and | control group. To identify the 
career decision making self-efficacy skill, a Pretest-Posttest Design with Nonequivalent Groups was used. Eighty 
participants, originating from 6 different regions of Thailand, were used as the participants in this research. Purposive 
sampling was selected from the upper secondary students, aged from 17-18 years old, all of whom attended the 
guidance summer camp at Chulalongkorn University, Bangkok during March 2017. Making up the demographics 
were the following categories, 26 males (32.5%) and 54 females (67.5%). Regarding geographical origin, there were 
participants from Central Thailand, as well as 23 (28.75%) from Northern Thailand; 17 students from the Northeast 
of Thailand (21.25%), Eastern Thailand (14.5%), and the South of Thailand (20%) 


2.2 Instruments and Procedures 


For data collection, the instruments used were comprised of an informed consent form, a lesson plan which 
was used during the group support activity of reflective e-portfolio development, and the Mahara tool (an open source 
e-portfolio system). The Career Decision Self-Efficacy scale (CDSE) measurements of Taylor and Betz (1983) were 
adopted so that the overall CDSE skill could be tested. Included in the CDSE skill are 5 sub-criteria; Self-Appraisal, 
Occupational Information, Goal Selection, Planning, and Problem Solving. The rating scale ranged from 1 (not 
confident), to 5 (most confident). Overall, 50 questions were used to determine the score variations, which were then 
divided into to a total of 5 points, the different points brackets are as follows: 1.0 to 2.5, which shows a low confidence 
level, 2.5 to 3.5, which shows that the level of confidence is moderate, and 3.5 to 5.0, which means the participant had 
a good level of confidence. Cronbach's alpha coefficient at a result of .966 was used to report the reliability of these 
figures. 


To collect the data, the participants were required complete the informed consent form voluntarily under the 
following conditions: 1) The participants are in the scope of research, 2) The participants receive sufficient information 
about the research and procedure, 3) The consent of the participants is voluntary, and 4) The results of the 
questionnaire will not result in any negative effects or residuals that may harm the respondents, cause stress, anxiety, 
or any negative impact on their personal well-being. After that, the participants were divided into the control and 
experimental groups. The control group served as a large support group (n=40) and the experimental group (n=40) 
was subsequently divided into sub-groups (5-8 members per group). This was done to evaluate the effects on support 
given, relative to group size, during reflective e-portfolio development. Pretest-Posttest assessments for all the 
participants was conducted by the researcher. Subsequently, the mean, SD and the statistical hypothesis testing of the 
p-value are revealed in the results. 


3 Results 
The aim of the present study was to compare the effects on support given, relative to group size, whilst developing 
a reflective e-portfolio. The reason for this was so that upper secondary school students had the opportunity to enhance 


their self-efficacy in career decision making. Self-efficacy skills displayed by the students, before and after e-portfolio 
development project, were analyzed using descriptive statistics. 
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Table 1 The comparison of mean score on each criterion, the total mean score of career decision making self-efficacy 
skill and the standard deviation in the control and experimental groups before and after experiment. 


<x = i n a n n 3S & 
a in ie 7.” a . oe 6 BE 
Criteria | B ° z | B £ B 2 B a vl 
n 
‘io io + “ie Be & 
Control Group (Large Support Group) 
M 3.59 3.73 3.65 3.81 3.48 Bee) 3.60 3.67 3.42 3.61 3.55 3.69 
SD 5 48 oT al ie] 39 AT A8 39 39 37 ee 
Experimental Group (Small Support Group) 
M 3.44 4.03 3.38 4.01 3.40 3.86 3.28 3.94 3.18 3.84 3.34 Bos 
SD wo .60 76 58 wi2 ag Ae! 61 78 ol 70 53 
Total Average 
M 3.51 3.88 3.51 391 3.44 Be) 3.44 3.80 3.30 3.72 3.44 3.81 
SD 61 6 68 me 58 el] 67 06 62 AT Ey) 48 


Abbreviations: Pre-Pre-test, Post- Post-test SA- Self-Appraisal, OI- Occupational Information, GS- Goal 
Selection, P- Planning, and PS- Problem Solving, total- total mean score of career decision self-efficacy skill 


The data was analyzed through the use of the criteria set, which were 1.0 to 2.5 for the low confidence level, 
2.5 to 3.5, which means level of confidence is moderate, and 3.5 to 5.0, which represents a good level of confidence 
in career decision making self-efficacy. It was discovered in the research that the control group (large support group) 
had an average score before the experiment, showing a good level of ability in every area except the goal selection 
criteria (M = 3.48, SD = .39), and the problem-solving criteria (M = 3.42, SD = .39). After the experiment, the average 
scores were good in all areas. In the experimental group (small support groups) the average score from before the 
experiment showed moderate levels Self-Appraisal )M=3.44, SD=.73(, Occupational Information )M=3.38, SD=.76(, 
Goal Selection )M=3.40, SD=.72(, Planning )M=3.28, SD=.79( and Problem Solving )M=3.18, SD=.78(, with a total 
average of )M=3.34, SD=.70(. The average scores after the experiment were good in all areas. 


Table 2 The comparison of Pre-Post mean score on each criterion, the total mean score of career decision making 
self-efficacy skill and the standard deviation in the control and experimental groups. 


M SD T P 
Control Group (Large Support Group) 
Pre - Post_SA 14 69 1.29 .20 
Pre - Post_OI 16 78 1.31 oll) 
Pre -Post_GS 16 50 2.03 .04* 
Pre - Post_P 07 .68 .68 49 
_Pre- Post PS nN 
Pre - Post_Total 14 54 1.68 10 
Experimental Group (Small Support Group) 
Pre - Post_SA 58 .88 4.19 .00* 
Pre - Post_OI .63 .86 4.66 .00* 
Pre -Post_GS 45 .89 3.24 .00* 
Pre - Post_P 65 93 4.41 .00* 
te POSE ES a eee eee eens eee ae 
Pre - Post_Total 59 .82 4.58 .00* 


*P<.05 


212 


It can be observed in the data shown in Table 2 that in the control group, the total mean score of career 
decision making self-efficacy was not different, at the level of .05 (Sig = 10). However, the research found a significant 
difference in the experimental group after the experiment (Sig = .00), where the scores were higher than before the 
experiment (M=0.59). 

In the sub-criteria: Self-Appraisal, Occupational Information, Goal Selection, Planning, and Problem 
Solving, the experimental activities carried out in the control group were not significantly different at the level of .05, 
except in the goal selection criteria, which was significant at .05 (Sig = .04). Conversely, in the experimental group, 
the post-experiment scores differed from the pre-experiment in all the criteria at the level of .05 (Sig. = .00), with the 
post-experiment scores being higher than pre-experiment: Sa=.58, Oi=.63, Gs=.45, P=.65 and Ps=.65 respectively) 


Table 3 The Pre-Post mean score on each criterion, the total mean score of career decision making self-efficacy skill, 
the standard deviation, and the P value in large and small support groups. 


M SD T MD P 

Large Group 

mn 3.73 aa 2.44 0.29 O1* 
Small Group 4.03 .60 
Large Group 

Gi 3.81 51 1.62 0.20 10 
Small Group 4.01 58 
Large Group 

a 3.65 ae 1.91 0.21 06 
Small Group 3.86 59 
Large Grow 

: Betree 3.67 a 2.11 -0.26 03* 
Small Group 3.94 61 
Large Group 
Small Group 3.84 51 
Large Group 
Small Group 3.93 53 


*P<.05 


Revealed in the analysis of data from Table 3 were the influences of implementing the reflective 
e-portfolio in social support groups, based upon the Activity Theory, and how it affected career decision making self- 
efficacy within the group. The Experimental group (small support group) was higher than the control group (large 
support group) at the level of .05 (Sig = .02), showing an average of 0.24. See figure 2 for the average score in each 
criterion of the large and small support groups after the development and implementation of a reflective e-portfolio. 


When considering the sub-criteria as 5 topics, it appeared that the scores for the sub-criteria of Self-Appraisal, 
Planning, and Problem-Solving were higher in the experimental group (small support groups), than the control group. 
Displaying significance at the level of .05 (Sig = .01, .03 and .03) On the other hand, the scores in Occupational 
Information and Goal Selection were not statistically, significantly different in the experimental group, at the level of 
.05 (Sig = .10, .06). 
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P GS 


—@=— Large Group 9 =@= Small Group 


Figure 2 shows the average score for each criterion of large and small support group after the development 
of a reflective e-portfolio 


4 Discussion 


The research results revealed the effects of group size on support provided during the development of the 
reflective e-portfolio. The size of a group should be considered as an important factor in the Activity Theory structure. 
It could also be seen that the results of this study indicate the division of labor in sub-social support groups (5-8 
members) was significantly higher than the larger group (40 members) (Sig at .05), which corresponds with the theory 
put forth by Albrecht and Adelman (1987, p. 62), who discussed the size of the support groups as being one of the 
factors to be considered when structuring social support. Trotzer (1997) argued that most counseling groups are made 
up of 10 members or fewer, and recommended a number of 6-10 members for groups of adolescents. Moreover, Pai, 
Sears, and Maeda (2015) also stated that the size of the subgroups positively affected the ability of the members to 
transfer knowledge, which supports the report of Bertucci, Conte, Johnson, and Johnson (2010). They claimed that 
small groups promote more academic support than solo work. Y. Cho et al. (2016) reported that while the post- 
experimental scores taken from the students’ training activities were not significantly different between large groups 
and subgroups, they did state that learners are more likely to need subgroup activities. In addition, subgroups usually 
have higher attention spans and return more feedback than large groups. However, COSKUN (2011) argued that in 
online collaboration, the size of the group does not influence the ability to work, but a large group size can actually 
help with the development of new ideas and creativity. There were many other factors found in this research which 
are yet to be considered. 


Regarding the structure of the activity system, it was could be inferred from the results of this research that 
the division of labor plays an important role in structuring the community whilst organizing activity theory based 
social support group activities. By defining the roles of responsibility in their assigned work using discussion, 
comment and feedback, both between and within groups, the expected results for the group members were accurately 
produced. The electronic portfolio was used as a mediating artifact during this process. Additionally, the results of the 
development activity can be seen in the summary of the research. These results are in line with the research conducted 
by Abidin, Uden, and Alias (2014), who stated an activity theory framework could be used to analyze the portfolio 
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development process, and this is also consistent with the findings of Yang and Wu (2013), who emphasized the 
increased level of self-efficacy skills through electronic portfolio development activities. Tammets and Laanpere 
(2014) reported that the e-portfolio could be a potential tool in recruitment, as a transitional instrument, used to help 
students move from academic institutions to professional institutions. Furthermore, Beckers, Dolmans, and 
Merriénboer (2016), carried out a systematic review in order to identify the influences of developing self-directed 
learning in an e-portfolio activity, leading to its integration into educational routines. 


The concept of collaboration amongst support groups, based upon the activity theory, is also consistent with 
social cognitive learning theory. Students were also able to reflect and inspect the functional division of labor within 
the social group, due to the e-portfolio activity. Bandura (1997) stated that behaviors are based upon interactions, 
made up of three components within a triadic reciprocality model: Behavior, Cognitive and other Personal factors, 
and Environmental. The composition of self-efficacy skills consists of the subjects’ own experiences, observation of 
others experiences, persuasion, and emotional stimulation. These are all the factors needed to develop self-efficacy 
skills in career decision making. Consistent with many of the researchers who have adopted this theory, Lent, Ezeofor, 
Morrison, Penn, and Ireland (2016) have emphasized the importance of the social support structure as one of the key 
elements which influences career decision making self-efficacy skills. In contradiction Hernandez, Oubrayrie-Roussel, 
and Préteur (2016), showed the negative relationship between social support, and attention and involvement in class. 
They stated that social support will predict students’ investment and interest in personal work, and success will only 
be achieved when the students pursue accomplishment and future goals. 


The observations carried out in this study conveyed that the development of reflective e-portfolios in small 
social support groups (5-8 members), based upon the activity theory, had a positive effect on overall career decision 
making self-efficacy. The relevance of using electronic portfolio in relation to time, support from mentors, as well as 
the relationship between learner and supervisor, was discussed by Tonni and Oliver (2013). Matthews, Karls, 
Doberneck, and Springer (2015) also stated that the use of an electronic portfolio provided an opportunity for students 
to contribute to the community through reflection on scholarships, in addition to being involved in community 
practice. Getting feedback from community members, receiving news feeds that have been screened by peers, and 
getting insights from their reflective thinking process are all part of this. Ontas and Tekindal (2015) also reported that 
the use of an e-portfolio contributes towards reflective thinking development. This relates to the work of Xu, Hou, 
Tracey, and Zhang (2016), who stated that self-efficacy skills in career decision making are enhanced by collecting of 
information on self and career processes. The information then leads to the decision made, through considering the 
value of careers, along with their educational factors. 


5 Limitations 


Both subgroups and large groups are both affected by group collaboration, as stated by Artinger and Vulkan 
(2016), which indicates that there are no differences between the levels of cooperation in large groups and subgroups. 
However, the researcher found that other factors which can affect this are the gender of the group members, but this 
cannot explain any differences in personality. These issues will be discussed in the research proposal. 


6 Conclusion 


The development of reflective e-portfolio activities along with social support leads to the integration of social 
roles, resulting in socially beneficial behaviors. During the reflective e-portfolio process, students reflected on 
themselves, in addition to giving feedback to their peers. All aspects of social support were combined throughout the 
reflection process, including socioemotional support, instrumental support, and informational support, which all 
contributed to resulting in higher self-efficacy skills in career decision making. This research determined the activity 
structure by defining the roles and responsibilities of community members. Included in these roles are: Subjects, 
Objectives, Community, Mediating Artifacts, Rules, Division of Labor, and Outcomes which could all be integrated 
into multidisciplinary teaching activities. However, when doing so, the structure and function must be clearly defined. 
Functional checks are performed by members within the group, and the evaluations, consisting of both formative and 
summative assessments, are completed in the using the same method. Finally, when completing social activities, the 
structure of the group should be considered, for example: the size of the group in the community can affect the level 
of interaction, function and participation of its members in a social structure. 
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Abstract 


This study examined the functioning of a group of instructional designers (IDs) in higher education through 
the lens of Communities of Practice (CoPs). The study particularly focused on whether and how the grouping of 
experienced and novice IDs operated as an effective CoP from the perspective of novices. The findings indicated 
that a group of IDs working in a midwestern university was able to cultivate a CoP within a clearly defined domain, 
a well-established community, and the shared practice with a specific body of knowledge. Particularly from the 
perspectives of novices, they highlighted the positive impact while participating in the CoP by contributing to their 
shared domain and defining who they are, developing expertise by interacting with experienced designers, and 
learning through different trajectories of participation. The rich description of this case study would further inform 
educators and practitioners in their efforts to improve the professional preparation and development for novice IDs 
in the higher education contexts. 


Introduction 


Research on instructional designers (IDs) has revealed common activities designers do and specific 
knowledge and skills they need to acquire (Tracey & Boling, 2013; Villachica, Marker, & Taylor, 2010). To prepare 
competent IDs, much attention has been paid to the formal training of IDs and professional development of expertise 
(Ertmer et al., 2008; Ertmer, York, & Gedik, 2009; Visscher-Voerman, Kuiper, & Verhagen, 2007). Among most 
Instructional Design and Technology (IDT) programs in the United States, it has become popular and conventional 
to immerse students in real-world authentic projects (Cennamo & Holmes, 2001; Knowles & Suh, 2005). The 
situated learning approach of infusing authentic projects within IDT programs has been recognized by many 
scholars as important in the efforts of preparing students for future practice (Tracey, Chatervert, Lake, & Wilson, 
2008; Leigh & Tracey, 2010). However, other scholars pointed out the gap between what IDT students learn in the 
classroom and how professional IDs approach design in reality (Kirschner, Carr, van Merrienboer, & Sloep, 2002; 
Leigh & Tracey, 2010; Visscher-Voerman et al. 2007). Instead of applying systematic models learned in academic 
programs, most designers in the real world conduct their work in a context-driven manner, which means they 
typically use different routes instead of strictly following a single model in the design and development phase 
(Visscher-Voerman & Gustafsonm, 2004). Therefore, scholars have argued that formal training is not sufficient for 
preparing novice IDs with the knowledge base and essential skills for professional instructional design (Tracey et 
al., 2008; Tracey & Boling, 2013; Yanchar & Hawkley, 2014). 

The importance of informal learning in the context of professional development has been increasingly 
acknowledged (Barton & Tusting, 2005; Yanchar & Hawkley, 2014). Such learning involves no formal curriculum 
or academic training but the “common, unstructured ways in which professionals become capable of performing 
their duties in the midst of professional practice itself’ (Yanchar & Hawkley, 2014, p. 272). Communities of 
practice (COP) are a form of informal learning and its effectiveness has been established through research for 
various professional groups (Barry, Kuijer-Siebelink, Nieuwenhuis, & Scherpbier-de Haan, 2017), including higher 
education, business schools, and coaching (Amin & Roberts, 2008; Barton & Tusting, 2005; French, 2011; Shams, 
2013). Particularly in higher education contexts, the concept of CoPs has been widely applied in science research 
groups (Creplet, Dupouet, & Vaast, 2003; Feldman, Divoll, & Rogan-Klyve, 2009; Maritz, Visagie, & Johnson, 
2013) and teacher education programs (Jimenez-Silva & Olson, 2012; Kaschak & Letwinsky, 2015; Sim, 2006). 
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Most studies showed promising results in terms of better preparing novice scientists and preservice teachers for 
developing professional expertise. 

Inspired by the positive impact of CoPs in higher education, this study explored how the grouping of 
experienced and novice IDs in a midwestern university cultivated a CoP; and in what ways this group of IDs 
operated as an effective CoP from the perspectives of novices. The implications of this study would help educators 
and researchers better understand how novice IDs navigate through the school-to-profession transition period as they 
become competent professional IDs. 


Literature Review 


The theoretical underpinning for this study resides in the literature of communities of practices (Wenger, 
1998; Wenger, McDermott, & Snyder, 2002), which refers to “groups of people who share a concern, a set of 
problems, or a passion about a topic, and who deepen their knowledge and expertise in this area by interacting on an 
ongoing basis” (Wenger et al., 2002, p. 4). In this section, we review the research literature on CoPs in science 
research groups and teacher education. 


Overview of CoPs 


The concept of CoPs is grounded in the sociocultural theories of learning and draws from the theory of 
situated learning proposed by Lave and Wenger (1991). The situated learning theory offers a holistic view of how 
learning takes place in the meaningful and authentic activities of a social group (Lave & Wenger, 1991). Lave and 
Wenger (1991) use the term “legitimate peripheral participation” to describe how individual learners gradually 
change their participation level in a community of practice based on their depth of experience. Ideally, as 
newcomers increase their engagement and become members of a community, the nature of their participation 
changes as do their identities in relation to other participants (Lave & Wenger, 1991). 

To associate practice and community and define a special type of community, Wenger (1998) described 
three dimensions of the relation by which practice is the property of a community: 1) mutual engagement (how and 
what things members do together and build relationships); 2) a joint enterprise (a set of problems and topics that 
members care about), and 3) shared repertoire (shared tools, artifacts, and resources that members create or share 
together to enact practice). Based upon their earlier work, Wenger, McDermott, and Snyder (2002) also developed a 
structural and practical model to guide community development. According to Wenger, McDermott, and Snyder 
(2002), a community of practice is a combination of three fundamental elements: a domain of knowledge, a 
community of people who are about this domain, and the shared practice that they are developing to be effective in 
their domain. In distinguishing COPs from other types of social structures, Wenger et al., (2002) defined COPs as “a 
very specific type of social structure with a very specific purpose” (p. 41). When all the three elements function well 
together, it makes a CoP an ideal knowledge structure that assumes responsibility for developing and sharing 
knowledge (Wenger et al., 2002). 


CoPs in Science Research Groups 


In higher education contexts, scholars have been examining how undergraduate and graduate students 
develop their research skills in the informal setting of research groups (Creplet et al., 2003; Feldman et al., 2009; 
Maritz et al., 2013). Feldman et al. (2009) examined how graduate and undergraduate students learn to do science by 
participating in research groups. They focused on the professors’ view of graduate education and coded the 
interviews with pre-conceived categories derived from the literature on graduate education, apprenticeships, and 
CoPs. Some of their results showed that: students’ learning was significantly influenced by the type of research 
group they joined (e.g. a tightly organized group would ensure students with more connections and interaction with 
several mentors); students could move along a continuum from “Novice Researcher” to “Proficient Technician” to 
“Knowledge Producer” if they received appropriate experience and guidance from their research groups operating 
like CoPs (Feldman et al., 2009). 

In an extended study, Feldman, Divoll, Rogan-Klyve (2013) focused more on the ways in which science 
and engineering students experienced their research education while participating in a research group. Their findings 
indicated that students participating in the research groups were not fully aware how others in the group were 
proactively teaching them to do research, and they gained both methodological and intellectual proficiency as novice 
scientists (Feldman et al., 2013). As their proficiency increased, their roles in the groups also changed from novice 
researcher to proficient technician to knowledge producer. Most importantly, each research group manifested the 
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characteristics of a CoP, which served as a site for students to take the role of apprentices by engaging in “legitimate 
peripheral participation” (Lave & Wenger, 1991). As Feldman et al., (2013) conceptualized, student learning in 
research groups is a set of learning trajectories that start with entering the group, then going through different levels 
of participation, and finally leaving as a novice researcher, proficient technician, or knowledge producer depending 
on their personal engagement. 

Examination of students’ learning experiences in research groups showed profound implications for the 
personal and professional growth of students (Feldman et al., 2009, 2013; Maritz et al., 2013). Especially in a tightly 
organized group, newcomers continuously received support from advanced students, developed greater cognitive 
and practical skills, and increased their engagement level along with knowledge growth, which becomes an essential 
part of professional socialization into their future professional practice (Davies, 2016; Hunter, Laursen, & Seymour, 
2007). 


CoPs in Teacher Education 


Much has been written in recent decades about the importance of facilitating genuine community and 
organizing learning in teacher education (Beck & Kosnick, 2001; Grossman, Wineburg, & Woolworth, 2001; Lee, 
Chen, Chang, & Yoneda, 2017). At the broader context of the program level, Lee et al. (2017) focused on how the 
sociocultural context of a graduate program influenced students’ perceptions of working with diverse students 
through the CoPs framework. Guided by the three dimensions of practice as the property of a community (Wenger, 
1998), they coded and analyzed their graduates’ perspectives about the program and organized their findings based 
on the three dimensions. They found that participants positively responded to how the program infused the goals and 
practices of multicultural education, and they indicated that their learning about multicultural education was 
influenced by the shared practices of the overall program instead of a particular course (Lee et al., 2017). They 
highlighted the usefulness of applying the CoPs concept for analyzing students’ perceptions of the program and 
noted that Wenger’s conceptualization helped them explain how the shared goals of the program were connected to 
their graduates’ perceptions (Lee et al., 2017). 

In addition to research examining CoPs at the program level, other scholars paid more attention to how 
CoPs were cultivated at the individual course level (Jimenez-Silva & Olson, 2012; Kaschak & Letwinsky, 2015). 
Multiple benefits were identified at both the level of undergraduate and graduate courses within the teacher 
education context. Kaschak and Letwinsky (2015) investigated how a CoP emerged unexpectedly in a middle-level 
mathematics and science methods course. As they found, project-based, service learning may act as “a pedagogy 
uniquely situated to develop communities of practice” (Kaschak & Letwinsky, 2015, p. 153). Moreover, they 
highlighted the importance of creating a sense of community in teacher education because CoPs may contribute to 
the means by which preservice teachers create new understanding about their practice (Jimenez-Silva & Olson, 
2012). Some researchers also paid attention to the teaching practicum as an important transition experience for 
preservice-teachers to acquire real-life teaching practice and form their professional identity as teachers (Sim, 2006; 
Sutherland et al., 2005). 

Overall, researchers agreed that creating a sense of community in the teacher education context is critical 
because it often leads to multiple positive outcomes on preservice teachers’ self-efficacy, which would potentially 
influence their future teaching practice (Beck & Kosnick, 2001; Grossman, Wineburg, & Woolworth, 2001; 
Kaschak & Letwinsky, 2015). The CoPs concept contributed to the ways that various institutions organized their 
teacher preparation programs or courses with a community emphasis. Such programs or courses designed around the 
CoPs concept helped encourage preservice teachers to build a culture of trust, acquire a sense of belonging that 
supports self-efficacy and contributes to effective instructional practice among preservice teachers (Jimenez-Silva & 
Olson 2012; Rauch et al. 2014). 


CoPs in Instructional Design 


As described above, CoPs has been effectively used in science research groups and teacher education. 
Preservice teachers, participating in a course or program designed around the concept of COPs, reported a better 
understanding of professional knowledge, practice, and identity as teachers, which could potentially enhance their 
instructional practice (Kaschak & Letwinsky, 2015; Lee et al., 2017; Sutherland et al., 2005). Similarly, science 
students participating in research groups operating as CoPs were able to improve their methodological and 
intellectual proficiency as novice scientists and gain professional socialization before entering into the real world 
(Davies, 2016; Feldman et al., 2009, 2013; Hunter et al., 2007). 


220 


The formal training of IDs has been organized in ways similar to that of teacher education programs. Both 
programs focus on educating students with fundamental theories and models that are essential to the field. However, 
most IDT programs in the United States do not provide real-world placements (e.g. teaching practicums in teacher 
education) or mentoring programs (e.g. research groups in science disciplines) that immerse students in authentic 
experience and apprenticeship learning before entering into the real world of professional instructional design. The 
results of the effectiveness of COPs in teacher education and science research groups suggest that COPs may be an 
effective method of preparing instructional designers. 

In the field of instructional design, little attention has been paid to the community and informal learning of 
IDs. Using grounded theory, Schwier, Campbell, and Kenny (2004) examined the professional identity of 
instructional designers via CoPs. Five full-time IDs with at least three years’ experience were interviewed. Their 
results indicated that: all participants indicated that their communities were born of convenience in an informal way; 
most of the participants learned about new theories, trends, and strategies more often from the interactions with their 
peers compared to any other means; most participants needed to mediate their own positions in the community from 
peripheral to experienced members; and they constantly shared solutions to design problems and built repositories of 
tacit knowledge through developing a community of practice (Schwier et al., 2004). Their study suggested the need 
to further investigate on how IDs interact and learn from each other by participating in a CoP. 


Purpose of the Study & Research Questions 


Traditional training in formal learning settings has been found to be insufficient for preparing IDs with 
essential knowledge or skills (Tracey et al., 2008; Tracey & Boling, 2013; Yanchar & Hawkley, 2014). There is a 
gap in the knowledge students gain via formal training in IDT courses and the knowledge and skills they need to be 
professional designers (Kirschner et al., 2002; Leigh & Tracey, 2010; Visscher-Voerman et al. 2007). To address the 
knowledge gap in IDs between formal training and professional practice, the potential of informal learning through 
CoPs warrants investigation. As the application of CoPs demonstrated promising results in preparing novice 
scientists and preservice teachers in higher education (Jimenez-Silva & Olson, 2012; Feldman et al., 2009), this 
study sought to further explore the functioning of a group of IDs in higher education through the lens of CoPs and 
expect to shed light on the professional preparation and development of novice IDs. 

This study focused on an online learning center at a large midwestern university that was comprised of 
professional full-time designers and part-time graduate students with varying levels of instructional design 
experience. The purpose of this study was to use this group as an intrinsic case to investigate how this group 
cultivates and operates as an effective CoP from the perspective of novices. The researchers believe this study is 
significant because it presents an alternative approach for preparing IDs by participating in a CoP. The research 
questions to be addressed in this study are: 1) Does this grouping of experienced and novice IDs, engaged in a 
higher education context for instructional design practice, cultivate a CoP? 2) In what ways, does this grouping of 
experienced and novice IDs operate as an effective CoP from the perspective of novice IDs? 


Methodology 


This study employed a descriptive case study approach that allowed the researchers to capture the 
complexity of real-life events and gain an in-depth understanding of a real-life case with multiple sources of 
evidence (Yin, 2014). As this study intends to provide a rich description of how a group of IDs operates like a CoP 
from the perspectives of novice IDs, a descriptive case study approach was identified as an appropriate mode of 
inquiry in this context to generate an overall picture of this group and then particularly focus on the experiences of 
selected individuals. 


Context of the Study 


A bounded system was selected as an intrinsic case in this study due to the unique nature of this case that 
might potentially manifest key characteristics of CoPs and help answer the research questions. This case is situated 
within a bounded time (one academic semester) and a restricted place — the Online Learning Center (OLC) at a large 
midwestern university which supports online and blended courses offered by the College of Engineering (CoE) and 
College of Liberal Arts and Science (LAS). As a support center primarily for online learning, OLC also provided a 
variety of services to faculty, staff, and students (e.g. technical support on educational technologies, consulting 
services on online pedagogies, and best practice resources). 
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This study focused on the Design and Delivery (D&D) unit consisting of three teams. Each team has a 
combination of full-time staff members and part-time Graduate Assistants (GAs). The number of hired GAs varies 
per semester depending on the workload of D&D unit. The main services provide by the D&D unit include: 1) 
supporting the development of the pedagogic design online and/or blended courses, 2) providing recording and/or 
hosting videos options, 3) providing information on making courses accessible to diverse learners, 4) 
troubleshooting technical issues in the courses, and 5) training instructors and/or teaching assistants to manage the 
day-to-day operations of courses. 


Participants 


As the second research question focused on the perspectives of novice IDs, a purposeful sampling strategy 
was used to select typical cases that might best answer the proposed research question (Creswell, 2013). Participants 
were recruited by e-mail from a pool of IDs employed at OLC at the time of conducting this study. The four 
participants cover a variety of backgrounds and experience as novice IDs: two of them are entry-level full-time staff 
members with 2-3 years’ experience, and another two are part-time Graduate Assistants (GAs) as newcomers with 
very limited professional experience in instructional design. 


Data Collection 


Three main data sources were collected in this study to generate a holistic picture of this unique COP for 
IDs, which included written reflections, individual interviews, and observations. 

Written Reflections. The participants’ reflections were collected through an online survey tool 
(Qualtrics.com) twice during the semester of Fall 2017. Each survey contained a similar set of four open-ended 
questions to collect participants’ self-reported information including a description of their practice, collaboration 
experience with colleagues, perceived level of participation, and other reflective thoughts. 

Individual Interviews. One semi-structured, individual interview was conducted with each participant at 
the end of the fall semester. Each interview was conducted in a private conference room on campus and lasted 
approximately 30-40 minutes. The interviews were used to explore their perceptions of and experience as novice 
designers. A set of questions was designed and asked in regard to their level of participation, sense of belonging, and 
collaboration experience. All the interviews were audio recorded and submitted to an outside service provider for the 
first-round transcription, and the researchers reviewed and revised in the second round for accuracy. 

Observations. The researchers observed the weekly group meetings of OLC’s D&D unit (around 1-1.5 
hour) and reviewed their previous meeting notes archived in Canvas. The focus of the observation was on gathering 
data about how the group functioned in general and how the participants interacted with others. Specifically, the 
observations intended to explore the nature of group interaction in general and examine the level of participation of 
each participant during the meeting. An observation protocol was developed to track individual participation level 
such as their frequency of speaking up, major topics discussed, how they responded to those topics, and how they 
interacted with other colleagues during the meeting. 


Data Analysis 


The data analysis followed theoretical propositions (Yin, 2014) framed by the theoretical framework (i.e. 
CoPs in this study). Specifically, the coding and analytic process were divided into two rounds and all codes were 
examined in the light of the study’s research questions. For the first round of coding, the researchers independently 
read the aggregated transcripts, generated explicit themes inductively, and constantly compare for similarities and 
differences, which allowed the researchers to reflect deeply on the contexts and keep open to multiple directions 
(Saldafia, 2016). For the second round, the researchers conducted an axial coding to group similar codes into each 
category (Saldafia, 2016), relabeled certain categories based on the CoP framework, and specified the data source 
and major codes for each category. To ensure the trustworthiness of the data, the researchers collected multiple 
sources of data for triangulation, used member checking for transparency, and provided a rich description of data 
collection and analysis procedures. The researchers employed two types of self-reporting data including reflection 
journals and individual interviews in order to minimize the bias from one single source. Particularly for the 
triangulation process, the coding process started from the individual interviews first, and then move to reflections 
with corresponding questions, and finally cross-checked with the observation notes. 
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Findings 


The findings are presented based on the order of the research questions. The initial research question was: 
Does this grouping of experienced and novice IDs, engaged in a higher education context for instructional design 
practice, cultivate a CoP? 


Observations of Researchers: Recognizing a CoP Cultivated by IDs 


According to the structural model of cultivating CoPs (Wenger et al., 2002), a community of practice is a 
combination of three fundamental elements: domain, community, and practice. The findings indicated the grouping 
of experienced and novice IDs was able to cultivate a CoP in an informal way. They have successfully established a 
specific domain comprised of instructional design theories and models, built a community for sharing information 
and learning together, and developed share practice with a specific body of knowledge. 

Domain. The data revealed that this group of IDs had a well-defined domain that creates a common ground 
that legitimizes the community by affirming the purpose and setting up the boundaries. Even though all the 
participants have different educational and professional backgrounds, they were all from educational disciplines and 
well prepared with a knowledge base that is essential for professional IDs. Their specific domain in instruction 
design created a common ground that not only brought them together but also facilitated knowledge building 
through contributing to the shared domain. 

Community. As one of the fast-growing fields, professionals in instructional design are expected to 
perform many roles and keep updating their knowledge and skills (Ritzhaupt & Kumar, 2015). This grouping of 
experienced and novice IDs helped to create the social fabric of learning that encouraged all members to update 
knowledge, share ideas, and foster interactions in collaborative efforts. Other than managing their primary business 
goal (i.e. supporting the online learning of two colleges), one of their ongoing activity was to keep updating their 
resource repository and studying their own practice. This secondary goal helped reinforce the significance of this 
group as a CoP and distinguish that from other social structures. 

Practice. As a well-established community with sustained interactions, the grouping of experienced and 
novice IDs also helped to develop a specific body of knowledge including a set of ideas, information, workflows, 
and frameworks that kept evolving with the community. the researchers also identified different trajectories of 
participation among the participants. As they came from a variety of experience level, they purposely chose a 
comfortable way of participation, which allowed a respective involvement for novices based on individual needs. 


Perspectives of Novices: Examining the Operation of an Effective CoP for IDs 


Next, the researchers will address the second research question: In what ways, does this grouping of 
experienced and novice IDs operate like an effective CoP from the perspective of novice IDs? 

Contributing to a shared domain and defining who they are. As the domain is essential to a community 
of practice (Wenger et al., 2002), participants indicated that their CoP has a well-developed domain that consists of 
key topics and issues that most IDs commonly care about. Based on the observations, most topics discussed in their 
online group chat and weekly group meetings were consistent. All the agenda items were proposed by group 
members instead of decided by their supervisor. Among those agenda items, most of them were very context- 
specific and solution-driven that encouraged everyone to share information and suggestions regarding their daily 
practice, including design challenges for a particular course/project, personal concerns for working and 
communicating with faculty, useful tips and tricks related to the LMS or other tools, plus any administrative issues. 

A divided contribution among the four participants has been noticed: both newcomers rarely proposed 
agenda items and passively participated in the discussion as they were still struggled to know the context and learn 
the convention; while two entry-level IDs contributed more actively in the meetings and felt more comfortable to 
provide others with insights and suggestions. While participating in those conversations centered on design 
challenges and solutions, participants found they got a better understanding of key topics and issues concerned the 
group and further extended their knowledge essential to the field. 

However, becoming an ID is more than being a skilled practitioner with sufficient knowledge but 
developing a sense of identity that belongs (or not belongs) to the professional community. While participating in 
the negotiation process of developing shared understandings of the domain, novice IDs also got the opportunity of 
defining or re-defining who they are, particularly in this case, referring to their perceived role and responsibilities of 
being an ID. 
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All the participants indicated that their perceptions changed over time as they became more familiar with 
their job and working context. Both entry-level designers identified themselves as a “supporting” role. Amy 
highlighted the technical aspect of her job and identified her role more as “instructional support”. Similarly, David 
also viewed himself as “more of a support role” but also highlighted other dimensions of the ID position such as 
“providing customer service” and “being diplomatic”. As described before in the context of the study, the Design 
team at OLC provided different types of grants for faculty to develop or improve online courses. David pointed out a 
specific situation for some grant courses that required IDs to be diplomatic while working with resistant faculty. 

For both newcomers, they also experienced a certain degree of perception change about being an ID. Beth 
as a former teacher originally identified the role of ID very similar to a teaching position but later realized another 
aspect of being a “project manager”, which was inspired by her supervisor in one of their weekly group meetings. 
Charles previously imagined that IDs mostly focus on “content development and production” but later felt 
overwhelmed by unexpected duties and then started to question his career aspiration and plan to look for 
opportunities in other fields. Even though all participants had slightly different understandings regarding their role 
and responsibilities as IDs, this group had a sense of collective image as David elaborated, 

I feel like we have a strong sense of our department who we are and we're kind of a badge of honor where 

we go on campus we just say we're with such and such. And most people understand who we are and where 

we are from. I think we all can identify as that and where it can get uniform (David, individual interview, 

11/06/2017). 

Overall, participants appreciated the involvement of a CoP with a well-defined domain consisting of key 
topics and issues in the field of instructional design. As they involved in more conversations about design challenges 
and solutions, they got the opportunity to further their knowledge by contributing to the shared domain. 
Furthermore, they also started to change their perceptions or rethink the role and responsibility of IDs as they 
became more familiar and competent with their job. 

Developing expertise by interacting with experienced designers. According to Wenger (1998), practice 
defines community in three dimensions: mutual engagement, joint enterprise, and shared repertoire. This section 
particularly explored how participants improved their professional expertise through each dimension above. 

Through mutual engagement, a CoP could become a tightly knit network of interpersonal relationships that 
could help novices get socialized into the group through close relationships and collaborations with long-standing 
members (1998). Both Amy and David as entry-level IDs identified the benefits of collaborating and interacting 
with colleagues as an essential part of the community. David believed that on a daily basis they “bounced ideas off 
each other”, which really helped him “think differently” and “have different perspectives”. And Amy addressed the 
collaboration part as a consistent part of her daily practice, 

Collaboration with my fellow instructional support person is a consistent part of my day. We discuss 

processes, meetings, follow-ups, tasks that need to be done. Almost everything we do is a collaborative 

effort. It allows us to stay consistent (Amy, individual interview, 11/14/2017). 

Another two newcomers also addressed the benefits of personal learning and growth by interacting or 
collaborating with more experienced colleagues, particularly referring to senior-level instructional designers in this 
study. Both of them highlighted the lessons learned from senior-level colleagues: As Beth mentioned in her first 
written reflection, she could always receive instant replies from other senior designers either in person or virtually 
(referring to emails and online group chat) whenever she needs some help; Charles pointed out in both of his 
reflections, it was “eye-opening” for him to “look at the forest instead of trees” and learn to design for 
“sustainability” as he worked with senior IDs. Below is one example Charles described, 

I was training with another senior instructional designer yesterday regarding one of my courses. Since she 

is more experienced than me and she recently worked on a similar task, she helped me a lot to consider 

bolts and nuts of the transition... I found that it turned out to be more effective training because I got to 

realize what I know and do not know more clearly (Charles, written reflection, 10/25/2017). 

A joint enterprise is the results of “a collective process of negotiation and efforts that reflects the full 
complexity of mutual agreement”, which creates relations of mutual accountability among members (Wenger, 1998, 
p. 77). Through participating in the negotiation process responding to their specific situations, participants identified 
themselves become more committed to their job by holding each accountable. As Amy described, their standard 
workflow has been frequently adapted by each designer based on their personal references, but they “constantly took 
the responsibility of discussing best practices and tried to figure out a better workflow and guideline” (Amy, 
Individual Interview, 11/14/2017). 

Shared repertoire as the third dimension refers to a set of “routines, words, tools, ways of doing things... 
that the community has produced or adopted in the course of its existence, and which have become part of its 
practice” (Wenger, 1998, p. 82). Based on our observations, this CoP for IDs has been using various strategies to 
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develop a shared repertoire including 1) a new hire guide that includes all must-know working process for 
newcomers, 2) an online resource repository that listed best practice and how-to documents for educational 
technologies, 3) an online group chat (Slack) that allows either individual or group chat for asking questions and 
sharing resources, and 4) ongoing research projects that investigated their design practices. Most participants 
appreciated the collective efforts on developing a shared repertoire, because it helped novices further expand their 
understanding of the job and develop their expertise as instructional designers. Especially for Beth and Charles who 
were new to the field, they had multiple sources of reference whenever they were looking for some tutorials, 
guidelines, and best practices. As Charles elaborated in his interview, he was looking for more direct supervision 
and support beyond the new hire guide created for internal training. Through observations, the researchers also 
recognized the usefulness of having a shared repertoire that served as a reference collection and helped new 
members catch up the history of the unit. However, the lack of additional support with direct contact has been 
identified as a shortcoming from the perspectives of novices. 

Learning through different trajectories of participation. The trajectory of participation is a phrase 
describing the direction that members in CoPs are moving, which is mediated by the amount of time getting 
involved and could be divided into five different types: peripheral, inbound, insider, boundary, and outbound 
(Wenger, 1998). New members will be naturally on the periphery given their struggles to learn the cultural 
conventions, which also suggested that CoPs are highly contextual and localized. Wenger, McDermott, and Snyder 
(2002) argued that the level of participation for each member within a CoP is voluntary with some tending to be 
more involved whereas others prefer remaining on the periphery. Members’ relations to a CoP could involve both 
participation and non-participation, and the mix of these two could shape their identities and impact their practice 
within a community (Wenger, 1998). As part of the learning process, participants presented different types of 
trajectories while participating in the CoP. 

For Amy and David who already worked in this group for 2-3 years, they identified themselves more as an 
“insider” and felt well connected to other members at the time of conducting this research. Both of them were 
following the type of insider trajectories that means their involvement does not end with full membership but may 
keep evolving with the community. 

For the two newcomers, they perceived their level of participation in different ways. As Beth also 
participated in other teams’ projects, she collaborated more frequently with other senior-level colleagues and 
identified herself as an “insider” who felt well supported and connected within the group. If Beth’s path was more 
likely following the inbound trajectories, Charles seemed to go for the outbound trajectories. As expressed in the 
interview, Charles saw himself more like an “outsider”, as he identified himself with insufficient knowledge and 
experience for participating in the conversations or offering recommendations to other. As Charles elaborated, 

If I think I have much knowledge and expertise in such area, then I will be insider. But I don't feel like it, 

then I'm okay with staying as an outsider. And I just think that is the right place at that stage. So right now, 

I think I'm kind of an outsider, which I'm okay with that (Individual Interview, 11/16/2017). 

It was important to notice that both newcomers experienced different trajectories of participation. Beth 
seemed to be moving into the inner circles of the community while Charles purposely chose to remain on the 
periphery. As Charles recognized the learning opportunity of being an “outsider”, he felt comfortable of staying on 
the periphery and identified this position as a strategy or a cover before moving forward. 


Discussion 


Many researchers argue that formal training is not sufficient for preparing IDs with the necessary 
knowledge base and essential skills for professional practice (Tracey et al., 2008; Tracey & Boling, 2013; Yanchar 
& Hawkley, 2014). The findings of this study revealed the potential of CoPs as an alternative approach for preparing 
novice IDs in an informal learning setting. By grouping with experienced IDs and participating in a well-established 
CoP, novice IDs experienced a smooth transition by quickly getting familiar with key issues and learning through 
different trajectories of participation. 

It is important to point out that informal learning will not automatically happen by grouping novices with 
experienced professionals together but requires some additional efforts. As indicated by prior research in science 
research groups, novice researchers could benefit most from tightly organized research groups as they could get 
frequent contact with and receive continuous support from proficient researchers (Feldman et al., 2009, 2013). This 
study also found that novice IDs gained considerable benefits for participating in a well-organized group operating 
like a CoP. The function of this ID group manifested key characteristics of CoPs, which could provide profound 
implications for other ID groups in similar working contexts. The first part of the findings already specified how this 
ID group working at OLC defined their specific domain in instructional design, built a community encouraging 
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learning and sharing, and developed shared practice consisting of specific knowledge and resources. For other ID 
groups in higher education institutions considering an organizational change for promoting informal learning, we 
would highly recommend further explore the concept of CoPs (Wenger, 1998) and its practical model (Wenger et 
al., 2002) as a guidance while re-organizing their groups for better preparing novices. 

As we probed further on the specific ways this grouping of experienced and novice IDs operated as an 
effective CoP from the novices’ perspectives, we identified three major themes regarding how participants perceived 
their experience. First of all, participants expressed clearly how they contributed to the domain of the CoP which 
consisted of key topics and issues concerned most IDs. It highlighted the importance of involving novices in either 
informal and formal conversations on essential topics during their transition stage. As novice designers typically 
have limited practical experience before getting into the professional field, their participation in a CoP with a well- 
defined domain could allow them to involve in the negotiation process of developing sharing understating and better 
relate theoretical knowledge to the practical realities (Sim, 2006; Sutherland et al., 2005). 

Moreover, as novices were exposed more to the domain of their CoP and involved in shared practice, the 
way they defined or re-defined themselves changed as well. It was evident that this ID CoP had a sense of group 
image. Only those who accepted the proposed group image would claim the membership of the CoP, while some (by 
choice or by necessity) failed to align with that may choose to leave the community. No matter which path they 
decided to go, we found each participant developed a better understanding of their job and their role as IDs. As 
revealed in prior studies, preservice teachers participating in a class or program designed around the COPs 
framework also reported a better understanding of professional knowledge and identity as teachers (Jimenez-Silva & 
Olson, 2012; Kaschak & Letwinsky, 2015; Lee et al., 2017). This finding pointed out the importance of further 
exploring the identity formation of novice IDs and examining how their professional identity would impact their 
design practice and performance. 

As novice IDs highlighted their growth of expertise by interacting and collaborating with experienced 
designers, this finding suggested the importance of grouping IDs with different levels of experience together and 
then pair them individually through a mentoring approach. Similar to how some novice researchers participated in 
science research groups operating as CoPs, they were found to gain both methodological and intellectual proficiency 
if received appropriate support and guidance within the group (Feldman et al., 2009, 2013). As one of the 
participants also indicated the need of acquiring more direct supervision and guidance, it pointed out the importance 
of providing more individualized support for novice IDs and shed a light on future research directions on peer 
support and individual mentoring for professional preparation. The last key theme also revealed that novice IDs 
could learn from different trajectories of participation. Similar to prior studies in science research groups, some 
trajectories may never lead to full participation while others could help a novice moving from a peripheral position 
of uncertainty to one of mutual engagement (Davies, 2016; Maritz et al., 2013). Through the lens of CoPs, it is 
important to recognize various types of trajectories that novices may experience and thus provide support and 
resources accordingly to accommodate individual needs. 


Conclusion 


This study particularly focused on whether and how a grouping of experienced and novice IDs ina 
midwestern university operated as an effective CoP from the perspectives of novices. All participants highlighted 
the positive impact while participating in the CoP by contributing to a shared domain and defining who they are, 
developing expertise by interacting with experienced designers, and learning from different trajectories of 
participation. The key findings recognized the cultivation of a CoP among a group of IDs and provided significant 
implications on how to better prepare novice IDs in similar contexts. For future studies, the researchers would 
recommend further exploration on how IDs interact and collaborate in an informal learning environment and how 
novice IDs formed their professional identity through different trajectories. By conceptualizing the nature of how 
novices learn from others and form their identity, it would reveal more insights and provide implications on how to 
better prepare novices for the professional fields. 
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Abstract 


Individualized mentoring programs have shown promising results in preparing faculty for technology use 
and integration, but very limited attention has been paid to what individuals gained from their experiences while 
serving as technology mentors for faculty. This study revealed insights on how graduate student mentors perceived 
the impact of a one-on-one technology mentoring program, specifically intended to identify the ways this mentoring 
program benefited their professional development and what recommendations they proposed for improving the 
program. Our findings indicated that graduate student mentors were able to improve essential skills for professional 
development through continuous collaboration and communication, develop a deeper understanding of technology 
integration in specific teaching contexts, and establish collaborative relationships with faculty mentees through 
individualized support. The results of this study have implications for educators and researchers working in similar 
higher education contexts to better design and improve their technology mentoring programs. 


Introduction 


In recognition of the importance and effectiveness of applying technology in school settings, there has been 
a history of emphasis on preparing teachers for technology integration (Ertmer & Ottenbreit-Leftwich, 2010; Ertmer, 
Ottenbreit-Leftwich, Sadik, Sendurur, & Sendurur, 2012). If teachers are expected to demonstrate meaningful 
technology use in their classrooms, it is important for them to see best practices modeled by faculty in teacher 
education programs (Brenner & Brill, 2016; Georgina & Hosford, 2009; Kay, 2006). That in turns requires teacher 
preparation programs and institutions adequately prepare faculty to effectively model technology use for preservice 
teachers (Chuang, Thompson, & Schmidt, 2003; Sprague, Kopfman, & Dorsey, 1998). 

Historically, teacher education programs have identified specific needs related to faculty mentoring and 
developed various strategies for faculty development in the areas of technology use and integration (Brush et al., 
2003). Among varying formats of faculty development models, prior research has reported that one-time workshops 
without individual support failed to meet specific needs of faculty (Chung et al., 2003). However, individualized 
mentoring has proven to be effective when addressing individual needs and supporting sustainable learning 
(Herring, Meacham, & Mourlam, 2016; Leh, 2005). 

Prior studies have focused on faculty members’ experiences as mentees while participating in various types 
of technology mentoring programs (Grove, Strudler, & Odell, 2004; Gunuc, 2015). Very limited attention has been 
paid to students’ experiences when serving as technology mentors for faculty (Leh, 2005; Kariuki, Franklin, & 
Duran, 2001). Present study revealed insights on how graduate student mentors perceived the impact of a one-on- 
one technology mentoring program on their professional development. Results from this study presents an in-depth 
understanding of the mentoring experiences and related benefits and challenges that graduate students faced while 
participating in a technology mentoring program specifically designed for faculty. This study provides implications 
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related to the design and development of similar mentoring programs in the future and helps identify the potential 
needs of graduate students related to technology professional development. 


Literature Review 


The word mentoring can be described in several ways, but a general definition offered by Hansman (2001) 
is the “intense caring relationship in which person(s) with more experience work with less experienced person(s) to 
promote both professional and personal development” (p. 28). As generally understood, mentoring refers to 
purposeful and organized interactions between more-experienced and less-experienced persons, where mentors 
provide instrumental knowledge and support (Eby, McManus, Simon, & Russel, 2000; Klasen & Clutterbuck, 
2002). The overall goal of mentoring is to help the less-experienced persons grow quickly in a transition stage and 
perform successfully in their professional and personal lives (Schunk & Mullen, 2013). 

Mentoring has been a popular approach to professional development in different contexts such as 
education, healthcare, and business (Bullard & Felder, 2003; Foster, 2001, Pamuk, 2009). Also, different types of 
mentoring are documented in literature and include such approaches as individual, peer-supported, group or team 
mentoring (Beyene, Anglin, Sanchez, & Ballou, 2002; Patton et al., 2005). In educational settings, the type of 
mentoring was typically determined by the purpose of the program or the needs and expertise of the participants 
(Campell & Campell, 1997; Jacobi, 1991). For example, experienced teachers supervised novice or beginning 
teachers to improve their classroom management, and experienced students helped less-experienced students while 
transitioning into a program. 

Particularly in the field of educational technology, various mentoring programs have been widely applied in 
K-12 settings to model the appropriate use of technology. One program in Southeastern Ohio paired instructional 
technology graduate students with eight teachers in a rural K-6 school to provide technology support (Kariuki, et al., 
2001). As their results reported, most participants noticed a transition from observers to co-learners, and all of them 
experienced a transformation in their understanding and use of technology (Kariuki et al., 2001). In a program 
combining web-based training and mentorship, Polselli (2002) found that K-12 teachers who received peer 
mentoring support reported a higher quality and instance of using technology as well as improved comfort levels 
with technology. Another program utilizing peer mentoring support for mathematic teachers combined workshops 
with individual mentoring sessions (Swan & Dixson, 2006). This study revealed that a mentor-supported model was 
effective in assisting teachers with additional support and provided opportunities to build an awareness of 
technologies available for use. 

Beyond the K-12 setting, different mentoring programs were also designed and implemented in higher 
education institutions with the aim of enhancing technology integration for faculty members. One program situated 
in a large Midwestern university paired nine graduate students as technology mentors for 19 faculty members (Smith 
& O'Bannon, 1999; Smith, 2000). This model featured a combination of pre-training sessions and one-on-one 
mentoring sessions. In semi-structured interviews, both the mentors and mentees highlighted an increase in 
technology use for faculty and students, and some graduate students also reported professional and personal 
development by gaining support and guidance from faculty members (Smith & O'Bannon, 1999; Smith, 2000). 

Similarly, California State University implemented a mentoring program that highlighted the combination 
of group (workshops & small-group meetings) and individual (mentors & technology helpers) training (Leh, 2005). 
Instructional technology graduate students were recruited, and each was paired up with a faculty member for 
individual mentoring support. Using formative evaluations, results indicated that both mentors and mentees 
benefited from the experience. It was reported that graduate students were able to raise their self-esteem, gain real- 
life experience, and benefit from the relationship and network built with faculty members while some graduate 
students also reported frustration with the difficulty of connecting with faculty (Leh, 2005). Another similar program 
initialized by Texas A&M University was the Technology Mentor Fellowship Program (TMFP) that matched 
undergraduate students with teacher educators with the purpose of modeling technology use. Results indicated that 
student mentors acquired substantial technology skills and communication skills while providing technology support 
to the teacher educators (Denton, Davis, Smith, Strader, & Clark, 2005). 

One-on-one mentoring programs have been found to be effective for assisting teacher educators with 
developing their technology skills in relation to the content they teach (Sprague et al., 1998). Both parties (mentor 
and mentee) involved in one-on-one mentoring programs identified multi-faceted benefits such as addressing 
individual needs with specific goals, building a mutual relationship, and fostering collaboration and learning 
community (Chuang et al., 2003). However, various challenges like time constraints, matching criteria, low 
motivation, and limited support and resources were also reported both from the mentors’ and mentees’ perspectives 
(Cox, 2005; Denton et al., 2005; Gunuc, 2015). 
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Even though previous studies have examined different types of mentoring programs (group vs. individual) 
in different settings (K-12 vs. higher education) or formats (online vs. in person), only a few studies explored the 
specific experiences and perceptions of the student mentors (Denton et al., 2005; Gunuc, 2015; Smith, 2000). Most 
rarely examined the specific teaching context and relationships of each mentoring pair. It is imperative to capture the 
individual experience and delve deeper into the specific contexts. This case study explored how graduate student 
mentors perceived the impact of a one-on-one technology mentoring program on their professional development. 
Specifically, we intended to identify in what ways this technology mentoring program impacted their professional 
development and what recommendations they proposed for improving this technology mentoring program. 


Theoretical Framework 


As participating in a mentoring program may not lead to any immediate behavior and attitude change, this 
study adopted the social learning theory (Bandura, 1977, 1986) to focus more on a process-oriented point of view 
and examine the interactions, relationships, and experiences associated with the mentoring program from the 
participants’ perspectives. 

Different from other traditional behaviorist theories, social learning theory argues that both internal and 
external factors play a role in the human learning process in different ways (Bandura, 1977, 1986). According to 
Bandura (1986), cognitive, behavioral, and environmental influences are not independent from each other but 
continuously affect human’s leaning in a reciprocal fashion. Therefore, individual learning has been viewed as a 
result of the reciprocal interactions between a person, behavior, and environment (Bandura, 1986). In addition, it 
emphasizes more on the role of social interaction and environment. It argues that learning occurs not only through 
actual performance but also observing other people’s actions and consequences as models (Bandura, 2001) Thus, 
modeling has been acknowledged as a fundamental and powerful way of disseminating knowledge and skills in 
various learning settings (Pamuk, 2008). 

Applying the concept into this study, each mentor/mentee (person) involved in the technology mentoring 
program may change their ways of using technology (behavior) which might impact their peers and others working 
in the same office or department (environment) as well. From the point view of social learning theory, it is important 
to be aware that each pair (the mentor and mentee) was not only influenced by each other but also potentially 
affected or could be affected by other pairs in the mentoring program or other peers in the community. 

The social learning theory was selected as a theoretical framework because: it focuses on learning from 
interacting and observing others that closely connect to the mentoring context of this study; it provides a lens to 
explain the process (instead of results) by keeping the focus on the interactions between or among the participants; 
and it directs us to pay attention to the contextual factors that may impact individual learning process that mentors 
and mentees could benefit from other sources beyond their individual mentoring pair. 


Context of the Study 


This study is situated in a graduate-level class during fall semester 2017 as part of the Curriculum and 
Instructional Technology (CIT) program in the School of Education (SOE). This faculty mentoring program in the 
SOE has had some type of presence in the program for 26 years. The main purpose of this faculty mentoring 
program is to bring graduate students together with faculty and to help faculty learn emerging technologies 
(Thompson, 2006). Moreover, it aims to assist faculty to meaningfully integrate these technologies into their 
curricula and become role models for using technology with their students. 

Most students enrolled in this class are doctoral and master students majoring in the CIT program. As part 
of a class requirement, all nine graduate students (mentors) enrolled were paired with faculty or staff (mentees) who 
teach undergraduate classes in the SOE. Mentees and mentors were paired by the course instructor at the beginning 
of the semester. The course instructor checked with each graduate student for their personal expertise and 
preferences (such as their prior teaching experience and prior knowledge and skills with various educational 
technologies). In the meantime, the course instructor also reached out to potential faculty and staff members and 
identified individuals with strong interest in participating in this one-on-one mentoring program. When possible, the 
instructor matched mentees with mentors based on the mentees’ needs and expectations as well as mentors’ skills 
and preferences. 

The overarching goal for the mentors was to help their mentees (i.e., faculty and staff) explore technologies 
relevant to their needs and interests, and thus encourage them to use these technologies in particular teaching or 
professional contexts. Therefore, mentors were expected to contact their mentees and set up an initial meeting to 
determine their own learning objectives and outcomes around the second week of the class. Other than the regular 
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face-to-face class meetings (2 hours per week) with other mentors and course instructor, each graduate student was 
required to meet with their mentee on a weekly basis for at least one hour using self-determined goals and outcomes 
to guide the mentoring process. In their initial meetings, each pair determined their regular meeting times and 
formats. Although each group generated their own goals and outcomes, one common goal for most mentoring pairs 
was to learn more about the Learning Management System (LMS - Canvas) because it was a transitioning semester 
for the entire institution from Blackboard to Canvas. 


Research Questions 


Multifaceted benefits and challenges such as mutual benefits, non-hierarchal relations, and sustainable 
relationships have been identified in the literature (Campbell & Campbell, 2000; Chung et al., 2003; Hasman, 2001). 
This study investigated how graduate student mentors perceive the impact of this technology mentoring program on 
their professional development. Specifically, this study was guided by the following research questions: 

1) In what ways, did this technology mentoring program benefit the professional development of graduate 

student mentors? 

2) What recommendations were proposed by graduate student mentors for program improvement based on 

the challenges they experienced? 


Methodology 


This qualitative study adopted a case study design to reveal the complex nature of technology mentoring 
practices through examining the perceptions of selected individuals and providing a rich description of the context. 
With the permission of the course instructor, the researchers sent out a recruitment email to all the potential 
participants (nine students enrolled in the class and the mentoring program), and seven students agreed to participate 
in this study. 


Table 1. Summary of Participant Profile 


Mentor Background Paired Mentee Technology Explored 
Alice Doctoral student, Faculty, science Explain Everything, Coggle, Popplet 
major in CIT methods 
Andrew Master student, Faculty, literacy Google Slides, Google Forms, TopHat, Canvas 
major in CIT 
Beth Master student, Faculty, science Flipgrid, iMovie, Camtasia, Arc, Canvas 
major in CIT methods 
Jacklyn Doctoral Student, Academic advisor Canvas, Box, Flipgrid, Top Hat, Kahoot, Nearpod, 
major in CIT Class Dojo 
Olivia Doctoral Student, Faculty, science BreakoutEdu, Dash&Dot, 3D Printing, Ozobots, 
major in CIT methods BlueBots, Makey Makey, Osmo, Code.org, Hour 
of Code, Scratch, Sphero, Little Bits 
Sarah Master Student, Faculty, literacy Google Drive, Google Slides, Google Forms, 
major in CIT Blackboard, Box, Lino, TitanPad, Wordle, 
LittleBirdTale, Flipgrid, Canvas. 
Zac Doctoral Student, Faculty, teaching Google Drive, Box, Blackboard, SimSchool, 


major in AgEd 


strategies 


ClassDojo, Canvas 


Multiple sources of data were collected after receiving approval from the university’s institutional review 
board. The data included: 1) an open-ended survey administered mid-semester, mentors were asked to report 
expectations, accomplishments, and experience, 2) individual interviews at the end of semester, mentors were asked 
to reflect on their experience and knowledge constructs, and 3) related technology artifacts created and/used by 


mentors. 


Our data analysis involved two rounds of coding. For the first round of open coding, we independently read 


the transcripts, generated explicit themes inductively, and constantly compared data for similarities and differences 
(Saldafia, 2016), which allowed us to reflect deeply on the contexts and remain open to different directions that 
could help us answer the two research questions. For the second round of axial coding, we tried to group similar 
codes into major categories as informed by our theoretical framework. To ensure the trustworthiness of this study, 
the researchers collected multiple sources of data for triangulation purposes, used member checking for 
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transparency, and provided a rich description of data collection and analysis procedures. All the researchers 
reviewed the major codes together and revised the codebook with iterative steps to ensure the inter-coder reliability 
throughout the data analysis process. Because the research team was also involved in the mentoring program and 
very familiar with most participants as personal connections, we were clearly aware of our potential bias and tried to 
interpret the results with caution. 


Findings 


Based on the analysis of the data, we focused on identifying in what ways, this technology mentoring 
program benefited the professional development of graduate student mentors. Research question | was stated as 
follows: Jn what ways, did this technology mentoring program benefit the professional development of graduate 
student mentors? Three major themes emerged as a result of data analysis and are summarized as follows. 


Improved Essential Skills for Professional Development 


As this program allowed each mentoring pair to determine their own learning goals and meeting format, 
each graduate mentor was involved in a one-on-one setting with continuous collaboration and communication that 
allowed them to build and improve a variety of skills essential for professional development. Among the seven 
participants, we noticed that five of them addressed the communication skills developed through the mentoring 
program. Most participants were actively communicating with their faculty mentees during and between their 
individual mentoring sessions. As Andrew said in the interview, “I think communication was key. I think ... trying to 
accomplish something in a short amount of time is challenging when you only meet once a week. I think the 
communication during the one-on-one meetings and in between was critical.” 

Other than communication skills, some participants also talked about other professional skills improved 
throughout this mentoring program including interpersonal, organizational, problem-solving, consultation, and 
leadership skills. Especially for those participants who were new to the CIT program, they needed to spend a lot of 
time on identifying the needs, exploring available technologies, and then providing appropriate solutions. Most of 
them emphasized the positive impact of this self-directed learning process, as they became accountable to their 
paired mentee and felt motivated to learn new technologies on their own. As Sarah pointed out in the interview, 

I think my problem-solving skill improved a lot after this mentoring experience. First, I needed to discover 

different technologies that my mentor may be interested in. Second, I needed to find strategies to deliver the 

information, deliver the technology to my mentee. Third, I also faced some challenges like some technologies 
we explored were also new to me. Before I taught those technologies to my mentee, I needed to learn them. 


Developed a Deeper Understanding of Technology Use and Integration 


Similar to other one-on-one mentoring programs, graduate mentors were paired with instructors teaching 
different subject areas, which allowed them to gain real-life experience and develop a deeper understanding of 
technology use and integration in specific teaching contexts. Although most participants were majoring in the CIT 
program, it was an essential requirement of the academic program for students to get familiar with a variety of 
educational technologies and understand how to use them effectively in educational settings. As Sarah reported in 
the mid-semester survey, “Through this mentorship experience, the most important thing I gained is to learn about 
different educational technologies. New Technologies are emerging every day. Especially as a CIT student, this 
experience helps me keep myself updated.” 

Participants were paired with faculty members teaching different grade levels and subject areas, so they 
were able to acquire first-hand experience of integrating technologies into specific teaching contexts in a meaningful 
way. Several participants described their rewarding experiences as getting familiar with the subject area taught by 
their mentees. For example, Andrew and Sarah found they explored different technologies that were useful in 
literacy classrooms, Olivia and Alice became more familiar with different coding and mind-mapping tools popular 
in a science classroom. Moreover, the mentors had the opportunity to apply different conceptual frameworks in the 
field of instructional technology. For example, most of them were using Technological Pedagogical Content 
Knowledge (TPACK) (Mishra & Koehler, 2006) to guide their mentoring practice and help their mentees focus on 
the interplay between and among content, pedagogy, and technology. As Jacklyn stated in the interview, 

I think the TPACK framework is a useful one for myself. I used that more like a conceptual framework for 

myself that I should the lead my mentee as a content expert into the center place. I think that it is an ideal 

place for a faculty or staff, um, to guide them into the center place of TPACK because as a faculty she is a 
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content expert. And I see myself as um, my expertise is on the technology and also on pedagogy. So, I think my 

Job is help integrate the three together and lead my mentee to go into the center of this place and consider the 

three components in a dynamic way. 

All graduate students met in a classroom on a weekly basis, and there most of them highlighted the 
importance of meeting regularly, sharing concerns, asking for help, and learning new technologies together. Several 
participants described similar scenarios in the interviews about how they supported each other. Especially for those 
who were new to the CIT program and not familiar with various educational technologies, they received useful 
recommendations and solutions from their classmates. 


Established Collaborative Relationships with Faculty Mentees 


Most participants also highlighted that the one-on-one setting made it easier for them to establish close and 
collaborative relationships with their faculty mentees in the program, which motivated both the mentor and mentee 
for continuous learning. As Beth elaborated in the interview, “I think the big thing is that it's just like that one-on-one 
piece. So, you get to build that relationship with each other and both of you benefit from that relationship.” Another 
example from Zac also addressed the importance of building relationship, “mentorship is not for you to apply what 
you want to set goals or just teach the mentee what she needs to learn. This is like mutual experience and mutual 
relationship, ask the mentee about her or his desires and needs.” 

As we delved deeper into how their collaborative relationship benefited their learning process, we noticed 
that each graduate mentor was utilizing different strategies based on their mentees’ needs in order to provide 
individualized support. Most of them started with modeling the use of new technologies and then encouraged their 
mentees to explore and practice on their own. Several participants also kept sustainability in their mind and always 
tried to direct their mentees to available support and resources either online or on campus. As they gained more real- 
world experience through providing individualized support, they also started to pay attention to the challenge of 
technology integration and kept reflecting on their practice. As Andrew described at the end of the interview, 

I understand kind of the practical elements of it, you know, ideally, you'd like to move very quickly and fast, 

but that's not the case a lot of times. You have to be strategic about what the things you do put in place. Um, 

so kind of pick and choose your spots and then understand that technology is like learning chunks. There is 
not necessarily like a sequential path, it follows this kind of like jumping around, but at the same time you are 
kind of filling in areas of need. 


Participants in this mentoring program encountered several challenges while working with their mentees. 
Research question 2 was stated as follows: What recommendations were proposed by graduate student mentors for 
program improvement based on the challenges they experienced? As a result of these challenges, specific 
recommendations were provided by the participants in survey responses and individual interviews, which was 
summarized as two major themes below. 


Improving the Organization of the Technology Mentoring Program 


Participants emphasized the importance of improving the organization of the mentoring program by 
revisiting the matching criteria, conducting the need analysis, and assigning homework for mentees. 

Revisiting the mentor-mentee matching criteria. Several mentors reported the challenges they faced as 
they came from diverse teaching backgrounds with their mentees and that made their relationships complicated at 
times. For example, Alice stated that her mentee had hoped to work with a mentor who had experiences from a 
similar subject area. She stated, “I think she (mentee) started out hoping that she would have a mentor that really did 
and all the content and that would have been a much more seamless process for her because somebody already knew 
the content was, you know, knew the methodology, the way that they approach science teaching.” Another mentor, 
Olivia, also pointed out the challenge she faced in finding the right content related technologies for her mentee, “as 
for the professional skills and backgrounds we were not good match, to be honest, because she was a science 
instructor and then I am coming from a language teaching and technology background. So, we tried to meet at the 
point of technology but since I was not that much familiar with the science contents/topics, it was kind of hard for 
me to find valuable technology for her.” 

Conducting a need analysis. Another challenge that some mentors faced was to determine the needs and 
goals that the mentees want to achieve. One of the mentors pointed out the importance of finding out the learning 
needs of mentees before the mentoring program starts. As Sarah elaborated in the interview, “Basically, except for 
Google Drive and Canvas, all other goals were what I established for my mentee, not what she asked. So, if I could 


234 


do this mentoring program again, I would send a survey to my mentee to ask specific questions about her 
expectations, for example, identify her needs, problems in her class, what she wants to engage/motivate her students. 
And then, I would go/explore those type of technologies instead of just randomly discovering some technologies.” 

Holding both parties accountable. There were several participants who emphasized the significance of 
assigning homework for their mentees because it held each other accountable week to week. As Sarah stated, “we 
decided to do ‘homework’ for each of us. Before our meetings, we went through all tasks because we had already 
decided like in the next meeting, we are going to explore some specific tasks and our homework was to watch some 
tutorials at home. We went through the learning modules in Canvas for teachers. And in our meeting, we just 
practiced the strategies we learned.” Some mentors thought that assigning homework would be extra work for both 
parties as Beth stated, “I had the feeling that she had so much on our plate already. She wouldn't have found that 
(assigning homework) beneficial, just would've been like a little more stressful... I think she was already like 
working on these videos and like recording them and doing the other stuff. So, she was still working towards her end 
goal even when | wasn't there.” Although assigning homework was thought to be extra work for mentees, it would 
worth further exploration for future studies to focus on specific strategies that could effectively hold both parties 
accountable and improve their learning experience. 


Creating a Learning Community for Sustainability 


Participants of this study provided several recommendations to create a sustainable learning community. 
One of the recommendations was to invite previous faculty mentees (participants in mentoring program in previous 
semesters) to the final meeting/presentation at the end of the semester, which was organized by the course instructor 
and invited all the mentoring pairs to share their experience and achievements. In that way, past mentees would keep 
themselves up-to-date with new technologies. As Andrew stated, “I think that they should invite all the past mentees 
to the final presentations because that would allow them to kind of connect with what new technologies are being 
used, what's being implemented, and allow them to kind of reconnect with their mentee and mentor experience.” 

Other participants also provided relevant recommendations such as building a virtual place for sharing 
information and new technologies and involving faculty mentees’ teaching assistants or colleagues into the 
mentoring process to collaboratively learn new technologies. As Jacklyn pointed out, “it would be nice to bring her 
TAs and other colleagues to join us. As I remember one session we met in October, her colleague next door who is 
also an academic advisor joined us to learn how to use Box and link that to Canvas. I think it will be a good way to 
make this learning outcome sustainable in the future, if we can encourage the faculty to bring their colleagues or 
TAs in. In that way, for the future, they can help each other and continuously learn new stuff.” 


Discussion 


Even though prior studies already addressed the mutual benefits for both parties involved in the technology 
mentoring program, they only summarized those benefits in general without little elaboration. For example, typical 
benefits mentioned include gaining support and guidance from faculty members (Smith, 2000), benefiting from the 
relationship and network built with faculty members (Leh, 2005), and acquiring substantial technology skills and 
communication skills (Denton et al., 2005). This study specifically identifies in what ways a technology mentoring 
program might benefit the professional development of graduate student mentors. Mentors in this study were 
involved in one-on-one collaborations that involved continuous oral and written communication, so they were able 
to improve essential professional skills such as communication, interpersonal, and organizational skills. Through 
being paired with instructors in different subject areas, they could develop a deeper understanding of technology use 
and integration in specific teaching contexts, and thus build a close, collaborative relationship with mentees through 
individualized support. 

On the other hand, various challenges including time constraints, matching criteria, low motivation, and 
limited support and resources were reported in previous studies (Cox, 2005; Denton et al., 2005; Gunuc, 2015). In 
the context of this study, participants provided specific recommendations based on the challenges they encountered 
in this program. We believe that addressing those environmental issues in the future would help leverage the quality 
of learning for all participants. As Cox (2005) indicated that the quality of matching the mentor and mentee affect 
the quality of mentoring process, our participants also highlighted the importance of revising the matching criteria. 
Some participants also suggested conducting a need analysis for faculty mentees before the mentoring program 
began. In reality, most of the mentees did not know what they needed or what technology they wanted to learn, so 
the graduate student mentors had to take the initiative and responsibility of identifying the potential needs and 
exploring new technologies. Prior research has also addressed the importance of creating a learning community 
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(Pamuk & Thompson, 2009), so this study specifically clarified different ways of facilitating a learning community, 
which suggests directions for future improvement of other technology mentoring programs situated in similar higher 
education contexts. 

As informed by social learning theory, that viewed individual learning as the reciprocal interaction of 
person, behavior, and environment (Bandura, 1977, 1986), we not only focused on individual participants 
themselves but also paid attention to the mentor/mentee and mentor/mentor interaction. It was important to notice 
that most participants reported personal growth by collaborating with their paired mentees and meeting with their 
classmates (i.e. other graduate student mentors) on a regular basis. Through modeling best practices and observing 
the implementation by their mentees, they gained more experience with using different technologies and developed 
a better understanding of technology integration in real-world teaching contexts. Another environmental factor 
impacting the mentors’ learning and growth in this program was their interactions with their classmates on a weekly 
basis for sharing information and concerns. It reminded us the importance of building a learning community to 
sustain the efforts they already established in this mentoring program. 


Conclusion 


In this study, we sought to identify specific ways this technology mentoring program benefited the 
professional development of graduate student mentors. In addition, important recommendations for improving the 
technology mentoring program were given. Overall, this one-on-one technology mentoring program received 
positive feedback from graduate student participants. Our key findings indicated the continued need for department 
and institution structures to be present for supporting such efforts because the individuals involved in a technology 
mentoring program might benefit in terms of professional development and knowledge acquisition. Future studies 
are highly recommended to expand the scope of this study and track the long-term impact of such technology 
mentoring programs on the professional development of both mentors and mentees. 
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